ML14178A430
| ML14178A430 | |
| Person / Time | |
|---|---|
| Site: | Robinson  |
| Issue date: | 12/28/1993 |
| From: | Peebles T NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML14178A428 | List: |
| References | |
| 50-261-93-34, NUDOCS 9401210109 | |
| Download: ML14178A430 (48) | |
See also: IR 05000261/1993034
Text
tpj
REG'(,
UNITED STATES
NUCLEAR REGULATORY COMMISSION
REGION II
101 MARIETTA STREET, N.W., SUITE 2900
ATLANTA, GEORGIA 30323-0199
U. S. NUCLEAR REGULATORY COMMISSION
REGION II
AUGMENTED INSPECTION TEAM (AIT) INSPECTION
Report Nos.:
50-261/93-34
Licensee: Carolina Power and Light Company
Docket Nos.:
50-261
License Nos:
Facility Name: H. B. ROBINSON UNIT 2
Inspection Conducted:
November 20-December 6, 1993
Team Leader:
,
_
__
_/,
/
3/k
Thomas A. Peebles, Chief
Date Signed
Operations Branch,
Division of Reactor Safety
Team Members:
M. E. Ernstes, Operator Licensing Examiner
E. D. Kendrick, Nuclear Engineer
S. M. Matthews, Quality Assurance Engineer
C. R. Ogle, Resident Inspector
B. H. Rogers, Reactor Engineer
J. E. Tedrow, Senior Resident Inspector
Approved by
22 1q3
1bert F. G' so
Direc r,
Date Signed
Division of ea tor Safety
9401210109 940105
ADOCK 05000261
G
EXECUTIVE SUMMARY
The objectives of the inspection were to determine the scope and the causes of
the events observed during the post refueling startup of H. B. Robinson Unit 2
and to evaluate the licensee's response to these events.
H. B. Robinson Unit 2 went critical on November 12, 1993. Criticality
parameters were within the expected range, and initial physics testing did not
reveal any core anomalies. On November 14, 1993, with the reactor at an
indicated power of 20 percent, a heat balance - done in response to management
questions about diverse power indications - showed that the actual power was
30 percent.
On November 16, 1993, flux mapping indicated core peaking factor problems.
These problems were confirmed by a second flux map. The licensee and the fuel
supplier (Seimens Power Corporation) discovered on November 18, 1993, that six
fuel assemblies had been misconstructed in that asymmetrically loaded,
burnable poison was incorrectly positioned in the core. The reactor had been
shut down November 17, 1993, in order to repair a steam leak in the secondary
plant.
The Augmented Inspection Team was chartered on November 19, 1993, and was
onsite November 20-24 and November 29-December 2, 1993. Additionally, members
of the inspection team were at the Seimens' Richland, Washington facility
.November
29-December 3, 1993, and a public exit meeting was held December 6,
1993.
The principal findings and conclusions of the Augmented Inspection Team were:
1. Licensee oversight and assessment of the fuel constructor, refueling
activities, startup preparations (including the calibration of nuclear
instruments and operator training), and the conduct of operations during
the startup were deficient.
2. Power range nuclear instruments had been mis-calibrated because of an
inadequate understanding of the core geometry, and the operators did not
diagnose the incorrectly reading power range instruments, although there
were sufficient indications available in the control room. Specifically,
lessons learned from an event at another plant in which power range
nuclear instruments were not reading correctly were not effectively
utilized to prevent this occurrence.
3. The plant operated with fuel having mis-positioned burnable poison. This
did not result in fuel damage, but damage could have occurred if the plant
had operated above 30 percent.
4. The six misconstructed fuel assemblies were the result of inadequate
fabrication controls and oversight by the fuel supplier.
5. The licensee's post event review and evaluation were adequate.
TABLE OF CONTENTS
Pagie
1.0
INTRODUCTION - AIT FORMATION AND INITIATION . ... . . . . . . . . .
1
1.1
Background
1.2
AIT Formation
2.0
EVENT DESCRIPTION ....... ... .. ... ... ... ....
2
3.0
MISCALIBRATED NUCLEAR INSTRUMENTS..........
......
. .
6
3.1
Review of root cause of miscalibrated nuclear instruments
identified during startup
3.1.1
AIT Findings and Conclusions
3.2
Assessment of the Adequacy of Nuclear Instrumentation
Calibration Procedures
3.2.1
AIT Findings and Conclusions
3.3
Assessment of operator performance relative to the nuclear
instrumentation miscalibration problem
3.3.1
Operator Interviews
3.3.2
Review of Operator Training
3.3.2
AIT Findings and Conclusions
4.0
CORE NEUTRON FLUX ANOMALIES . ...... ... .. ... ... ..
15
4.1
Assessment of root cause and safety significance of
the core neutron flux anomalies with regard to fuel and
Technical Specification limits
4.1.1
AIT Findings and Conclusions
4.2
Assessment of the cause and extent of fuel manufacturing
errors at Siemens Power Corporation-Nuclear Division Fuel
Manufacturing Facility
4.2.1
AIT Findings and Conclusions
4.3
Assessment of the extent and effectiveness of fuel
assembly verification at Siemens Power Corporation
Nuclear Division
4.3.1
AIT Findings and Conclusions
0
III
4.4
Assessment of the extent and effectiveness of fuel
verification at the site
4.4.1
AIT Findings and Conclusions
4.5
Assessment of the Siemens Power Corporation-Nuclear
Division analysis of the core neutron flux anomalies
4.5.1
AIT Findings and Conclusions
4.6
Assessment of the licensee's oversight of Siemens Power
Corporation- Nuclear Division fuel analysis and Quality
Assurance programs
4.6.1
AIT Findings and Conclusions
5.0
BROKEN FUEL INSPECTION TOOL DURING REFUELING . . . . .......
24
5.1
Determination of the root cause of the broken fuel
inspection tool event
5.1.1
From the On-site Inspection
5.1.2
From the Inspection at Siemens Power Corporation
Nuclear Division
5.1.3
AIT Findings and Conclusions
5.2
Determination of the effectiveness of licensee oversight of
contractor fuel handling activities
5.2.1
AIT Findings and Conclusions
5.3
Assessment of the adequacy of Siemens Power Corporation
Nuclear Divisions Quality Assurance program for the manufacture
of special
fuel tools
5.3.1
AIT Findings and Conclusions
5.4
Assessment of the effectiveness of Siemens Power Corporation
Nuclear Divisions program for notifying licensees of known
deficiencies in either hardware or services provided
5.4.1
AIT Findings and Conclusions
6.0
ASSESSMENT OF THE LICENSEE'S INVESTIGATION OF THESE EVENTS . . . .
34
6.1
Assessment of the effectiveness and thoroughness of the
licensee's investigation of these issues
6.1.1
AIT Review of the licensee's Nuclear Instrumentation
Miscalibration Review Team Assessment
6.1.2. AIT Review of the licensee's Robinson Fuel Loading
Investigation Team Assessment
7.0
EXIT MEETING .. ..... ....
....
...... ....
..
35
APPENDIX A - AIT CHARTER AND SUPPLEMENT
APPENDIX B - THE LICENSEE REVIEW TEAMS
APPENDIX C -
EXIT ATTENDANCE
FIGURE X -
FUEL INSPECTION TOOL
FIGURE Y - GADOLINIUM FUEL ASSEMBLY AS DESIGNED VS AS-BUILT
FIGURE Z - RECOMMENDED ASSEMBLY TO USE FOR NUCLEAR INSTRUMENTATION CALIBRATION
III)i
1.0 INTRODUCTION
1.1
Background
During the restart of H. B. Robinson Unit 2, following the cycle 15 refueling
outage, which refueled the unit with the Cycle 16 core, the NRC was aware of
the following sequence of events:
DATE
TIME
EVENT
11/12/93
12:14 am
Commenced reactor startup.
11/12/93
6:19 am
Reactor critical.
Estimated Critical Position
met.
11/14/93
7:00 am
Reactor at the point of adding heat.
11/14/93
8:00 am
Intermediate Range nuclear instrumentation NI-36
bypassed.
11/14/93
9:22 am
Operations realized that there was a reactor
power mismatch:
30 percent actual; 20 percent
indicated.
11/14/93
9:00 pm
Loose Parts Monitoring System discovered
deenergi zed.
11/14/93
Evening
Site team formed to review startup.
11/16/93
4:00 am
Flux map indicates abnormal peaking factors.
(Core design problem.)
11/16/93
6:15 pm
Leaking weld identified (at FW-13B).
11/16/93
8:40 pm
Second flux map confirms peaking factors.
11/17/93
1:13 am
Started unit shutdown to repair FW-13B.
11/17/93
Morning
Licensee management review team on-site to
investigate.
11/18/93
Afternoon Licensee and fuel vendor decide that core
misloading is the cause of the core peaking
factor problem.
1.2
AlT Formation
On November 19, 1993, senior NRC managers concluded that events surrounding
this startup warranted further independent evaluation; an Augmented Inspection
Team was formed, and a Confirmatory Action Letter was issued by Region II. A
detailed charter was developed to guide the team. In addition to the above
events, loose parts had been identified during fuel handling activities in the
T
wspent
fuel pit.
Inspection of this item was included in the charter (the AlT
III2
Charter is Appendix B).
The team began its inspection on site on November 20,
1993.
2.0
EVENT DESCRIPTION
The following is the detailed sequence of events associated with the
November 12, 1993, startup until Hot Shutdown was reached on November 17,
1993. The team found no major disagreements from the licensee's sequence of
events. A detailed narrative begins at paragraph 3.0.
DATE
TIME
EVENT
11/12/93
12:14 am
Shut reactor trip breakers, commenced
reactor startup per Procedure GP-003.
11/12/93
12:20 am
Commenced Procedure EST-050, Refueling
Startup Procedure. (Control Rod
withdrawal then dilute to criticality.)
11/12/93
6:08 am
Deenergized Source Range Nuclear
Instruments, and NI-32 hung up at 45
cps.
11/12/93
6:18 am
Start Up Rate meter pegged high. Stopped
Procedure EST-050. Power stabilized at
108 amps in the Intermediate Range.
11/12/93
6:19 am
Reactor critical.
11/12/93
11:49 am
Cleaning of Start Up Rate meter selector
switch complete.
11/12/93
1:00 pm
Recommenced Procedure EST-050
11/12/93
2:00 pm
Commenced low power physics testing per
Procedure EST-050.
11/13/93
11:40 am
Completed low power physics testing.
11/13/93
1:28 pm
Reactor at one percent.
11/13/93
2:07 pm
NI-44 returned to service following
Procedure EST-50.
11/14/93
12:52 am
NI-35 Out Of Service for setpoint
changes.
11/14/93
3:12 am
NI-35 returned to service.
3
11/14/93
3:14 am
NI-36 Out Of Service for setpoint
changes.
11/14/93
3:55 am
NI-36 returned to service.
11/14/93
6:39 am
Latched main turbine.
Operator Distractions
Left main turbine stop valve bypass
won't open due to isolated instrument
air valve, IA-3221, shut. Unable to
bypass around and equalize across left
stop valve.
Watchstation turnover.
Turbine rolling approximately 200 rpm
due to leakage past the governor valves.
Operators concerned over Procedure
GP-005 precaution to minimize time
turbine below 200 rpm.
Turbine vibration alarms occurred.
11/14/93
7:47 am
Turbine valve trip test. Number two
intercept reheat valve does not indicate
closed.
11/14/93
7:51 am
Turbine relatched.
11/14/93
7:54 am
Turbine at 1800 rpm.
11/14/93
8:08 am
Unit on line. Power escalation
commenced.
Operator Distractions
Sluggish voltage regulator response.
No Mega Volt Amperes Reactive indication
on gauge board.
Load dispatcher reports telemetry
failure.
All four turbine governor valves
indicate shut.
11/14/93
8:15 am
Intermediate Range NI-36 is bypassed at
Nuclear Instrumentation cabinet to
preclude reactor trip.
- I
4
Operator Distractions
Low level in "A" Steam Generator.
No feed flow indication on A & C Steam
Generators.
Steam flow greater than feed flow
alarms.
Feed Water Heater Alarms.
Swap of electro-hydraulic oil pumps due
to A pump not unloading.
Main steam reheater vent to condenser
valve (FCV-1334) indication problem.
Balance of plant operator required to
hold valve switch on gauge board to
open.
System engineer reports turbine
vibrations recorded .in
control room
twice field reading.
Condensate header low pressure alarm due
to condensate pump recirculation valve
FCV-1446 hung open.
System engineer identifies one of four
generator H2 coolers isolated.
11/14/93
8:42 am
Main Feedwater regulating valves in
auto. Power escalation stopped to
stabilize reactor power.
11/14/93
8:57 am
Reactor power stabilized at 20 percent
as indicated by Power Range Nuclear
Instrumentations
11/14/93
9:02 am
Electrical operator questions indicated
Nuclear Instrumentation power based on
turbine first stage pressure equates to
approximately 26 percent power.
11/14/93
9:22 am
Engineering Technical Services manager
questions difference between reactor
power indicated by Power Range Nuclear
Instrumentations and net Mega Watts
electric. Operators estimate 30 percent
reactor power based on loop delta Ts.
5
11/14/93
9:40 am
Generator volt-ampere (mega volt amperes
reactive) knife, switch found open.
Closed switch and restored mega volt
amperes reactive indication.
11/14/93
10:26 am
Initial calorimetric data per Procedure
OST-10 indicates reactor power is 30.26
percent.
11/14/93
10:44 am
Operations Manager and Engineering
Technical Support Manager notified of
initial calorimetric results per
Procedure OST-10
Reactor engineer
support requested.
11/14/93
10:47 am
Decreased reactor power to less than 30
percent.
11/14/93
11:01 am
Procedure OST-10 complete. Calculated
reactor power is 30.26 percent.
11/14/93
12:20 pm
Regulatory affairs notified of potential
TS 3.10.7.1 violation for exceeding
three percent/hr ramp rate immediately
following refueling.
11/14/93
2:00 pm
Procedure EST-53 indicates actual
reactor power is 30.03 percent.
11/14/93
2:57-3:12 pm
Instrumentation and controls personnel
recalibrated Power Range Nuclear
Instrumentati ons.
11/14/93
9:00 pm
Loose parts monitor discovered disabled.
When placed in service, alarm received
on primary side of Steam Generator C.
11/14/93
11:30 pm
Noise from potential loose part on Steam
Generator C subsided.
11/16/93
4:00 am
Operations notified that flux map
results identified peaking factors that
require additional analysis.
11/16/93
6:10 am
NI-35 out of service to reset high flux
trip and rod stop.
11/16/93
5:33 pm
NI-35 high flux trip and rodstop reset.
No retest due to procedural problems and
plant conditions.
- 6
11/16/93
8:10 pm
Feedwater leak identified on main
feedwater pump "A" discharge. Second
flux map indicates core flux
irregularities.
11/17/93
12:35 am
Feedwater leak traced to weld crack for
feedwater drain valve, (FW-13B).
Crack
growing.
11/17/93
1:13 am
Unit shutdown commenced at one
percent/min per Procedure GP-006.
11/17/93
1:35 am
Station output breakers opened.
11/17/93
4:14 am
Reactor shutdown.
11/17/93
8:39 am
NI-35 failed Procedure OST-001.
11/18/93
11:16 am
Completed Shutdown Procedure GP-006.
3.0
MISCALIBRATED NUCLEAR INSTRUMENTS
3.1
Review of the root cause of the miscalibrated nuclear instruments
identified during startup.
During the plant startup on November 14, 1993, licensee personnel discovered
that power range nuclear instrumentation indicated power readings were
approximately ten percent lower than actual reactor power. The licensee
attributed the cause for this discrepancy to be the effect that the new
reactor core had on the power range nuclear instrument indication and an
improper understanding of core geometry considerations. During the cycle 15
refueling outage, the licensee installed a very low leakage core. The new
core load not only reduced the neutron flux present at the reactor vessel, but
also reduced the neutron flux which reached the nuclear instrument detectors
located on the outside periphery of the vessel. T *o
account for this change in
the neutron flux, the intermediate range and power range nuclear instruments
were recalibrated to adjust the previous cycle instrument currents to
predicted newcycle instrument currents.
The team reviewed the licensee's startup testing which was performed following
the refueling outage and the associated nuclear instrumentation work packages
which documented the calibration of the nuclear instruments with particular
emphasis on the intermediate range and power range detectors. In addition,
the licensee's investigation into this event was reviewed.
Procedure FMP-002, "Nuclear Instrumentations Post Refueling Adjustment
Determination," was implemented by the licensee to quantify the projected
impact on the nuclear instrumentation by the new core loading. The team
verified the licensee's calculations in this procedure for the new power range
100 percent currents and the intermediate range rod block and high level trip
setpoints.
The team reviewed Work Request WR 93BMZw1 and verified that the
100 percent currents were implemented intothe power range channels.
7
Westinghouse issued a letter to the licensee on March 16, 1988, providing
guidance on nuclear instrumentation concerns when implementing a core design
change which reduces the neutron leakage from the core. This letter was
initiated because of core design changes made at the licensee's Harris
facility. The letter recommended a correction factor which was calculated
from the average relative power of four fuel bundles, which comprised the
complete outer diagonal nearest the power detector. This guidance was not
incorporated at plant Robinson even though corporate licensee personnel were
aware of the Westinghouse letter which was implemented at the Harris
facility. Instead, licensee personnel developed the calculations contained in
Procedure FMP-002 based on previous cycle performance data which agreed more
closely with historical core data. These calculation utilized the two nearest
outer diagonal fuel assemblies and a third inner assembly (second diagonal)
for relative power comparisons.
The licensee determined that their method was in error following communication
with the fuel vendor, who stated that the outer fuel assemblies contributed
approximately 16 percent each to the flux indicated on the detector while the
inner assembly contributed only 2-3 percent. The incorrect methodology was
not detected during previous.startups because previous core loadings
fortuitously had fuel assemblies with similar relative power loaded in the
inner position and on the periphery of the outer diagonal, thereby canceling
any mathematical averaging errors. The new, low leakage core on the other
hand, specified that a much higher average relative power assembly be loaded
- in
the inner position which differed substantially from the outer diagonal
assemblies. When the averaging calculations were performed for the new core
load, the error in methodology caused a discrepancy between predictions of
approximately 90 percent. Based on the information from the licensee's
vendors, the team concluded that the root cause for the nuclear instrument
miscalibration was due to the incorrect methodology used in calculating the
power range currents. This was confirmed by licensee personnel who calculated
the predicted power range indication using the correct methodology. These
calculations indicated that when actual power was 30 percent the correctly
calibrated power range instruments would have indicated approximately 38
percent, which would have been conservative and acceptable.
Also during this outage, both of the source range and intermediate range
nuclear instrument detectors were replaced due to aging. The methodology used
in the intermediate range calculations agreed between Westinghouse and the
licensee and therefore the calculations for intermediate range rod stop and
high level trip setpoints were unaffected.
As part of the startup test program, Procedure EST-050, Refueling Startup
Procedure, was used to establish the high power reactor protection system
trip setpoint at 45 percent. The TS limit for this setpoint is 109 percent
power. The action to reduce this setpoint was taken in response to a similar
nuclear instrument miscalibration which occurred in December 1998 at the
licensee's Harris facility which is similar in design to the Robinson plant.
The team reviewed work request WR 93HUK001 completed on November 9, 1993,
which implemented the 45 percent trip setpoints. The team considered this
action by the licensee to be very beneficial which would have limited a
potential power excursion. The team calculated that a reactor trip would have
- II8
occurred once actual power reached approximately 67.percent, which is far
below the technical specification limit. The team considered the conservative
action to reduce the high power trip setpoint after refueling outages to be a
program strength which helped reduce the potential consequences of this event.
3.1.1
AIT Findings and Conclusions
The team determined that improper methodology for predicting Power
Range currents was used by Robinson. Inadequate corporate/site
oversight and communications contributed to this use of improper
methodology. The licensee corporate fuels staff was aware of the
Westinghouse recommended method for predicting currents, its basis,
and its implementation from prior experience at the Harris plant.
As a safety precaution, the setting of the Power Range High Power Flux
Trip had been set at 45 percent vice 109 percent prior to startup,
this would have limited any power increase to less than an allowed
value. However, the core flux anomalies coupled with the power range
nuclear instrument misalignment would have resulted in high neutron
flux in localized areas of the core if power had reached the indicated
45 percent.
3.2
Assessment of the adequacy of station nuclear instrumentation
calibration and refueling procedures.
The licensee's nuclear instrument calibration procedures were reviewed as well
as work packages which were performed on the nuclear instruments during the
plant startup. Both of the source range and intermediate range nuclear
instrument detectors were replaced. The team discussed the procedure guidance
with licensee personnel and compared the procedure scope with control wiring
diagrams and the system technical manual to determine completeness. The
following procedures were reviewed:
Nuclear Instrument System Source Range
Nuclear Instrumentations Pulse Amplifier NM-101,
Attenuation, Discrimination and High Voltage Power Supply
NQI1
Nuclear Instrument System Intermediate Range Channels NI-35
& NI-36
Nuclear Instrumentations Power Range Channel NI-41, NI-42,
NI-43, and NI-44
- PIC-107 Power Level Indication at the Power Range
e PIC-109 Nuclear Instrument System Over Power Trip High Range
Adjustment for the Power Range Flux Detectors
- PIC-110 Nuclear Instrumentations Intermediate Range (NI-35 & NI-36)
Compensating Voltage Adjustment and Loss of Compensating
Voltage Alarm Adjustment
.The
team found the guidance provided in the licensee's calibration procedures
to be adequate and closely agreed with the system technical manual.
However,
9
the team found a few deficiencies in the data sheets provided in Procedures
LP-704, LP-705, and PIC-109. The data sheets were considered by the team to
be confusing since procedure sections were not specifically identified for
data recording. Further, the team noticed that acceptance criteria was not
included on the data sheets. This information was only included in the body
of the procedure. Although this practice did not prevent successful
completion of the procedure, it hampered supervisory review of the completed
package. Licensee personnel had already identified this matter and
appropriate procedure changes were planned to upgrade the data sheets.
From a review of the work packages, the team identified implementation
problems. The intermediate range nuclear instruments were not calibrated with
the new rod stop and high trip setpoints calculated by Procedure FMP-002 until
after the reactor was critical and at the point of adding heat on November 14,
1993. This situation was contrary to the requirements of Procedure EST-050,
step 3.10, which documented that the Nuclear Instrumentation adjustments per
Procedure FMP-002, had been completed prior to taking the core critical.
The
team discussed with the responsible individual, why this requirement was
initialed as completed without the appropriate adjustments being completed.
The person responsible indicated that due to miscommunication with the
maintenance technicians, he believed that the adjustments had been completed.
The licensee's investigation had also identified this deficiency. The team
reviewed the safety significance of this situation. The intermediate range
rod stop and high level trip setpoints were not required by the licensee's
.technical
specifications. Since the setpoints which were present at the time
of reactor criticality were set at the old cycle values and were lower than
the new predicted currents, startup with the old setpoints was considered to
be conservative by the team.
In addition, the work packages (WR 93-AJTB1, WR 93-AJBG2) associated with the
replacement of the intermediate range NI-35 and NI-36 detectors, were reviewed
by the team. Typically the licensee replaces the source range detectors every
outage due to aging effects, and the intermediate range detectors are likewise
replaced at the same time due to location. No discrepancies were identified.
3.2.1
AIT Findings and Conclusions
Written procedures were generally considered to be adequate.
Deficiencies were noted in some data sheets; for example, acceptance
criteria and tolerance bands were not specified, and procedure
sections were not specifically identified. The licensee had already
identified these issues, and the procedures were included in an
upgrade program but had not been completed prior to startup.
Implementation problems were noted in establishing the Intermediate
Range Nuclear Instruments' high level trip and rod stop setpoints
prior to criticality - they were not done until the point-of-adding
heat. Also, source range NI-32 channel was not recalibrated following
detector replacement even though the procedure and technical manual
recommends that this should be done. Procedural and work controls
were lacking; however, the old setpoints were found to be conservative
by the team.
10
3.3
Operator Performance relative to Nuclear Instrumentation
miscalibration problem.
3.3.1 Operator Interviews
At 12:14 a.m. on November 12, 1993, following the completion of Refueling
Outage 15, the licensee commenced a reactor startup. The startup and
subsequent power escalation were performed in accordance with three
procedures:
General Procedure, GP-003, "Normal Plant Startup From Hot
Shutdown to Critical;"
Refueling Startup Procedure, EST-050; and General
Procedure, GP-005, "Power Operation."
Procedure GP-003 established the
initial conditions for the startup. Procedural control was transferred to
EST-050 for initial criticality and zero power physics testing. The
escalation of power into and through the power range was accomplished with
Procedure GP-005. This sequence and other key events of the startup are
documented in Paragraph 2.0. At 9:22 am on November 14, 1993, with power
stabilized at 20 percent on the nuclear instruments, the Manager of
Engineering Technical Support questioned the apparent mismatch between power
range Nuclear Instrumentations and net Mega Watts electric. Estimates of
reactor power by the operators from loop delta Ts, indicated that power was
close to 30 percent. A subsequent calorimetric calculation confirmed this
estimate and the power range Nuclear Instrumentations were set to thirty
percent. The increase in power to 30 percent caused a violation of technical
.specifications
in that the 3 percent per hour rate of power rise limitation
between 20 percent and 100 percent of reactor power specified in Technical
Specification 3.10.7,was exceeded. The actual rate of power increase was
approximately 10 percent in a 15 minute period. A flux map performed at 30
percent power indicated flux tilt and anomalous power levels. The crew
maintained power at 30 percent while efforts were made to resolve the flux
anomalies. A second flux map provided similar results. Following the
discovery of a secondary side steam leak, a reactor shutdown was commenced on
November 17, 1993.
The team attributed the mismatch between the actual power and the Power Range
Nuclear Instrumentation readings to be one result of an inadequate calibration
procedure. This conclusion and its basis are discussed in Paragraph 3.1.
The team reviewed the startup to assess operator performance relative to the
nuclear instrument miscalibration problem. This review consisted of
interviews of control room operators, as well as reviews of instrument traces,
plant computer printouts, the completed startup procedures and the shift
supervisor and reactor operator logs.
Each watchstander interviewed, cited prevention of a plant trip as his major,
if not primary function. None of those interviewed verbally attached a
similar significance to monitoring instrumentation for failure or
inaccuracies. This focus on preventing a trip may have resulted in key
individuals concentrating on a limited range of plant parameters resulting in
11
ineffective oversight by members of the watch section. This reduced the
potential for earlier identification of the power mismatch. Site management
was not aware of this focus and had not provided adequate direction to cause
the shift to be also be observant of the overall plant conditions.
The shift supervisor expressed concern prior to the watch over an Intermediate
Range Nuclear Instrumentation reactor trip. He experienced one during a
startup at Robinson in 1988. Adding to this concern, was an E-mail message
from a reactor engineer sent the previous evening, warning of potential
problems in the response of the Intermediate Range Nuclear Instrumentation
during the power ascension. The memo addressed Intermediate Range Nuclear
Instrumentation detector setpoint adjustments and the potential for achieving
the intermediate range rod stop (20 percent current equivalent on the
Intermediate Range Nuclear Instrumentation) prior to satisfying the P-10
interlock (10 percent on the Power Range Nuclear Instrumentations). The memo
did not mention the adjustments to the Power Range Nuclear Instrumentations
performed during the outage. In essence, the memo led the operators to
believe that the Power Range Nuclear Instrumentations would be a more reliable
indicator of reactor power than the Intermediate Range Nuclear
Instrumentation. Furthermore, the discussion in the memo on satisfying the
intermediate range rod stop at 20 percent prior to satisfying the P-10
interlock at 10 percent, reduced the potential for the operators to question
an apparent 10 percent mismatch between the Intermediate Range Nuclear
Instrumentation and Power Range Nuclear Instrumentations. The memo reinforced
- the
crew's concern for a trip on the intermediate range high flux prior to
satisfying the P-10 interlock. For these reasons, a shift supervisor,
supplementing the crew, was assigned to monitor the Intermediate Range Nuclear
Instrumentation to ensure that it did not exceed the trip setpoint prior to
bypassing the trip functions.
This assignment prevented this individual from
maintaining an overview of the plant during a portion of the power ascension.
Several watchstanders cited difficulties in control of steam generator level
and Tave as significant distractions during the startup. Review of computer
printouts indicated that severe oscillations occurred in the steam generator
levels until automatic level control was established. These difficulties were
in part complicated by the lack of feed flow indication at low power levels on
two of the steam generators. One reactor operator was dedicated to control of
feed flow and the steam generator water levels. The SRO was also involved
with the steam generator water level control as this was recognized as having
a high potential of causing a reactor trip.
The team was unable to determine categorically if these difficulties in plant
control were more severe than in prior startups or severe enough to mask the
power mismatch. However, the team noted that even if the entire efforts of
the three control board operators were directed at plant control problems,
three shift supervisors and a shift technical advisor were still present to
perform oversight and overview of the startup.
Throughout the interviews, watchstanders identified distractions as detracting
from their efforts during the startup. The major distractions that occurred
at key points in the startup are included in the timeline discussed in
Paragraph 2.0. While it is obvious that any distraction would impact operator
12
performance during the startup, the team was unable to assess the severity of
this impact. The team did note that the operator's self-assessment of the
impact of these distractions covered a broad spectrum from severe to minimal
impact.
A thorough, pre-evolution brief was not conducted coincident with watch relief
immediately prior to the power escalation. The team identified that the crew
did not review the precautions in Procedure GP-005 in detail prior to assuming
the watch. Step 4.22 of Procedure GP-005 was added as a corrective action to
a similar event which occurred at Shearon Harris in 1989. This step states:
"During power ascension, all indications of reactor power level should
be monitored and compared. Periodically, indications such as core
delta T and turbine first-stage pressure should be compared to NI
indications. If all indications do not agree within 5 percent,
Reactor Power should be stabilized and an OST-010 performed."
Application of this precaution would have identified the Power Range Nuclear
Instrumentation mismatch earlier in the startup. Since this need to compare
indicated power with other indications of power was not duplicated in the body
of the procedure, this precaution was not brought to the operators' attention.
Immediately prior to the power escalation, the on-shift crew was relieved with
the turbine latched and rolling. Precaution Step 4.15 of Procedure GP-005,
limits the time that the turbine can be operated below 520 rpm. Concerns with
violating this precaution were cited by at least one watchstander as providing
an impetus for relieving the watch and commencing the power escalation. The
desire to accomplish this expeditiously may have contributed to the less than
adequate pre-evolution preparation.
Finally, the team noted that poor communications contributed to the failure of
the watch section to diagnose the power mismatch. Although the reactor
engineer's E-mail memo, discussed previously, contained specific direction for
the operators to stop the power ascension if the 20 percent current equivalent
rod stop was achieved prior to the P-10 interlock, the memo was not routed
through the Operations Manager. A copy was provided to the Operations
Manager; however, he stated that he was not aware of its existence prior to
the power ascension. A second example of a communications failure was
evidenced when two operators stated that they had raised questions over the
accuracy of the indicated power range after power was stabilized at 20
percent. These concerns resulted from inconsistencies in turbine first stage
pressure and indicated neutron power were made prior to the questioning of the
net Mega Watts electric reading. However, these observations were not
communicated to the entire crew for resolution and were not reflected in the
operator logs.
The team concluded that sufficient information was available to control room
operators to permit diagnosis of a deviation in indicated power and actual
power prior to exceeding an actual power level of 20 percent. An analysis of
control room instrument traces and plant computer records by the team revealed
that several instruments indicated that the operators should have questioned
the Power Range Nuclear Instrumentation readings. Further, these indications
- II13
were available to the operators prior to exceeding an actual core power level
of 20 percent. Specifically, loop delta Ts, turbine first stage pressure, and
one of the Regulatory Guide 1.97 wide range power level instruments, all
provided clear indication that actual power was greater than that indicated by
the Power Range Nuclear Instrumentations.
The team concluded that watchstander distractions during the power escalation
contributed to the failure to detect the Power Range Nuclear Instrumentation
miscalibration. These distractions were primarily the result of the
following: a focus by key watchstanders on preventing a reactor trip that
overrode maintaining adequate oversight; difficulties in controlling certain
plant parameters; and equipment malfunctions.
The team also concluded that the small magnitude of the Moderator Temperature
Coefficient during the startup increased the necessity for reactivity control
through frequent rod motion. Additionally, these power fluctuations added to
the difficulty in controlling steam generator water levels.
Contributing factors included distraction of watchstanders, an inadequate pre
evolution brief, watch relief with the plant in an other than stable
condition, and poor communications.
3.3.2 Operator Training
.In
1989, the licensee's plant Harris experienced a similar event in which the
Power Range Nuclear Instrumentations were discovered to be miscalibrated
during a reactor startup. Industry Document SOER-90-003 described this event
and corrective actions.
The training on this industry event, at another
facility owned by the same licensee, did not adequately prevent its recurrence
at Robinson. The lessons learned from this event were covered only once
during requalification training after the Harris event. It was not
incorporated into any of the requalification training given after the first
year. The training method did not adequately reflect the problems encountered
in the Harris event; for example, the simulator scenarios did not challenge
the operators with a Power Range Nuclear Instrument that was indicating low
during startup conditions. The scenario had the operating crew detecting the
inaccurate indication at 90 percent power by doing a procedurally required
calorimetric. This did not reinforce the concept of monitoring diverse
indications of power during a startup.
The initial license training program contains a lesson plan with an excellent
description of the Power Range Nuclear Instrumentation miscalibration event at
Harris. This was not used in requalification training. The only operator on
shift who had recently been licensed, received this training but could not
recall the details of it.
Several operators stated that the startup training did not adequately reflect
the actual plant startup. The crew members participated in about four hours
of simulator startup training. This startup contained no malfunctions. The
simulator feedwater controls and instrumentation respond with such precision
that the operators get little training on what is experienced in an actual
startup. There were no distractions which would have required crew
14
prioritization or coordinated oversight. Additionally. the startup training
did not mention the new low leakage core. This would have been helpful in
informing the operators of the expected response by the Nuclear
Instrumentations.
Another contributing factor identified from the operator training was the lack
of management involvement in the training. There was no Operations Department
interface to relay their expectations to the control room operators.
3.3.3 Discussion of the Bypass of Intermediate Range Instrument Nuclear
Instrumentation-36 High Flux Trip.
At 8:15 am on November 14, 1993, the level trip switch on intermediate range
instrument NI-36 was placed in the bypass position to defeat the intermediate
range high flux trip. This trip occurs at a nominal intermediate range
current equivalent to 25 percent reactor power. The trip is not considered in
the plant's safety analysis and is not required by technical specifications.
When the trip was bypassed, NI-36 was indicating approximately 7.6x10-5 amps
with the trip set at 1.3X10 4 amps. This action deviated from Startup
Procedure GP-005 which required that the trips be defeated using the
Intermediate Range "A" and "B" Logic Trip Defeat push-buttons on the gauge
board. This can only be accomplished when the P-10 interlock is satisfied;
i.e., 2 of 4 power range instruments indicated greater than 10 percent. When
questioned on the appropriateness of this action, the watchstander stated that
this evolution had been pre-briefed with the shift supervisor, that the P-10
interlock was satisfied prior to the action, and that the action was taken due
to the tolerances assigned to the trip setpoint. As described by the
watchstander and others, a trip on intermediate range high flux had occurred
during a previous startup with little warning.
From a review of plant computer printouts, the team noted that at the time the
NI-36 high flux trip was logged as defeated, the P-10 interlock was satisfied.
The team also noted th&t the intermediate range high flux trips were correctly
defeated in accordance
th Procedure GP-005 approximately 1 minute later at
8:16 a.m. Based on their review, the team concluded that though the action
was not in accordance with the startup procedure, the safety significance of
this deviation was minimal.
3.3.4 AIT Findings and Conclusions
Operators had sufficient indications to detect the difference between
power range indications and actual reactor power.
The crew's focus on trip prevention overrode maintaining adequate
oversight.
The operating crew did not trust their Intermediate Range Nuclear
Instrumentation indications.
0,
There was not a pre-evolution brief to adequately emphasize
precautions or expectations of the operating crew.
15
Start up training did not reflect the actual plant start up.
Training on the Harris event was not effective in preventing the
occurrence of a similar event at Robinson.
The potential consequences of this event were minimized since the
initial goal of the power ascension was to stop at about 30 percent.
The power ascension was stopped early with readings of the power range
nuclear instruments indicating about 20 percent, but documentation of
other readings at that time found the power to have been actually at
30.3 percent.
Startup Procedure GP-005 did not prevent reoccurrence of the Harris
event. Although a precaution to monitor diverse indications of power
during power ascension was added, this was not read or implemented by
the crew during this start up. There were no expected values for Mega
Watts electric, delta T, or turbine impulse pressure listed in the
procedure to flag problems at 10 percent and 20 percent power. This
lack of guidance was a significant contributor.
Management did not make their expectations clear as to control room
watchstander duties and responsibilities.
4.0
CORE NEUTRON FLUX ANOMALIES
4.1
Assessment of root cause and safety significance of the core neutron
flux anomalies with regard to fuel and technical specification limits.
Early in the core design turnover process from the fuel vendor to the
licensee, the licensee's Nuclear Fuel Services group noted some computer input
discrepancies (from the INCORE Code) which the fuel vendor then addressed.
After the core was delivered, the final approval was then given by the
licensee's Nuclear Fuel Services for the site to load the core and conduct
startup physics and power ascension tests.
Two in-core maps were taken at 30
percent power between November 14 and 16, 1993. The first map was number 698,
and the latter one was number 699.
An error in different computer core design data (using the PDQ Code) was
detected after the 30 percent power in-core flux maps had been performed. The
fuel vendor's PDQ computer input deck that was part of the in-core analysis
conducted on November 16-18, 1993, did not include two items:
ITEM 1.
The gadolinium rod overlays for the latest core reload batch
(This computed higher predicted individual assembly powers in the
gadolinium rods than if the gadolinium overlays had been used, and
consequently other rods assembly powers were predicted lower.)
ITEM 2.
The six misconstructed asymmetrical gadolinium fuel
assemblies (This item was, of course, not known at the time.)
16
A new computer in-core analysis was rerun the first week of December with the
PDQ computer input corrections.
The new computer input deck (with ITEM 1 fixed and presuming the core
was loaded as-designed with properly configured assemblies) was run by
the licensee's Nuclear Fuel Services for the number 698 and 699 maps
taken at 30 percent power. Results were calculated as follows:
-
The new computer data calculated the predicted relative assembly
power for the original expected design.
-
The in-core maps, numbers 698 and 699, showed the as-measured
condition of the core at 30 percent power and calculated the as
measured relative assembly power.
-
The differences were then computed on an assembly by assembly
basis and showed what should have been the results on November 16,
1993.
This would have been the information available to the site and the
licensee's Nuclear Fuel Services to decide if the as-measured core
contained any anomalies that were significant, and whether the
misconstructed assemblies would have been detected.
This analysis found that in-core map indications would have been present and,
if the original computer data had been correct, would have led engineering to
detect the misconstructed assemblies when the first maps were analyzed.
Another computer input deck (with ITEM 1 fixed and the core loaded as
built, with the six misconstructed assemblies) was run by the
licensee's Nuclear Fuel Services for the number 698 and 699 maps.
This calculated the actual November 16, 1993, F-delta H and showed
that the technical specification limit was exceeded by less than 0.5
percent.
-
The new computer data calculated the predicted relative assembly
power in the as-built core.
-
The in-core maps, numbers 698 and 699, showed the as-measured
condition of the core at 30 percent power and calculated the as
measured relative assembly power.
-
The differences were then computed on an assembly by assembly
basis and simulated what could have been the results on
November 16, 1993, if the core had been loaded correctly. This
allowed the in-core map to be analyzed for any other
discrepancies.
No other anomalies were observed in this analysis and discrepancies are not
expected to be seen after the core is reloaded with the misconstructed
assemblies in the proper locations.
4.1.1 AIT Findings and Conclusions
The.team reviewed the Cycle 16 analysis, that was run with the correct design
parameters, and found:
17
The F-delta H technical specification limit was exceeded by less than
0.5 percent on one in-core map and the other map did not show a
technical specification violation. This shows that the limit may have
been exceeded by a small amount.
The core radial tilt resulting from the misconstructed fuel assemblies
was observable once the corrections were made to the computer data.
The Cycle 16 fuel had operated well within the Departure from Nuclear
Boiling limits and no damage to the fuel should have occurred.
Reactor coolant chemistry data analysis also showed no fuel damage.
4.2
Assessment of the cause and extent of the fuel manufacturing errors at
the Siemens Power Corporation-Nuclear Division fuel manufacturing facility.
On November 18, 1993, during cycle-16 plant start-up, it was determined that a
manufacturing error had occurred and six misconstructed fuel assemblies had
been installed in the Robinson core. The fuel assemblies had been built 90
degrees out of the correct orientation because incorrect load map drawing
information had been entered into the manufacturing computer system. Two
subsequent Quality Control overchecks failed to detect the error.
Fuel assembly manufacturing was controlled by the Bundle Assembly Data Logger
computer system. The computer program compiled information associated with
the fuel assemblies and provided technicians with manufacturing control
instructions which indicated which rods to place in which position of the fuel
assembly and in what sequence to do so.
Fuel assembly information was loaded into the computer program by a Production
Control Clerk (clerk) who performed this function to assist the Production
Control Specialist (specialist) who typically performed the task. The clerk
used the specialist's identification and password to access the computer to
perform this task. The fuel vendor indicated that the common use of the
identification and password was not prohibited and that separate
identification was not set up specifically for the clerk because he performed
a variety of tasks. The process of entering the fuel assembly information
into the computer program was not specified by procedure, and the clerk had
only an informal document available for guidance. In addition, the clerk
performed the task of loading the fuel assembly information into the computer
program at a remote computer terminal, between other employees' work stations,
with little work space to accommodate the required documentation.
The computer program was configured so that a specific set of information
(header information) appeared at the top of the screen. The header
information included the fuel assembly (bundle) item (part) number and drawing
number, the load map item (part) number and drawing number, the manufacturing
order number, and the project title. When entering the information for fuel
assembly item number 140148, the clerk entered an incorrect load map drawing
18
number 308181 (the correct load map drawing number was 308180 for fuel
assembly item number 140148). When entering the information for fuel assembly
item number 140150 the clerk entered the incorrect load map drawing number
308180 (the correct load map drawing number was 308181 for fuel assembly item
number 140150).
The fuel vendor determined that the clerk had worked on fuel assemblies 140148
and 140150 during the same session at the computer program computer terminal
and that it appeared that the load maps and the attached insertion sequences
were swapped between the two packages.
The computer then provided a series of prompts to allow the clerk to enter the
location of the fuel rods within the fuel assembly. The documents used to
load in the fuel assembly information included the parts list (a reviewed and
approved design document), the load map drawing (a reviewed and approved
design document), and the insertion sequence (an informal document which
indicated the order in which the assembly table placed fuel rods into the fuel
assembly, a manufacturing document). The clerk defined a set of "find
numbers" used to identify rod types and then entered the insertion sequence
using the find numbers. This information was obtained from the load maps and
insertion sequences which were incorrectly specified for fuel assembly numbers
140148 and 140150. As a result, the computer program was loaded with design
information and manufacturing instructions which would place the fuel rods in
incorrect positions when the fuel assemblies were manufactured.
After entry of the fuel assembly information was completed, the computer
program produced a bundle proof map (the header information, a matrix of find
numbers representing the fuel assembly, and the find number definitions) for
each fuel assembly. The bundle proof maps for fuel assembly numbers 140148
and 140150 were verified by Quality Control Engineering by comparing the
matrix of find numbers and the find number definitions to the load map drawing
specified in the header information for each fuel assembly. However, the
Quality Control Engineering person did not verify that the load map drawing
number listed in the header information was correct (it was incorrect for fuel
assemblies 140148 and 140150).
The team reviewed the procedure governing the
quality control activities, "Fuel Bundle Map Verification," Revision 0, dated
August 3, 1990, and noted that it did not clearly specify the basis for the
checks but only specified that the checks be made against a "hard copy."
Following completion of the-manufacturing process, the computer program
provided an as-built bundle map of each fuel assembly which listed the part
number (or no load) which was located in each coordinate of the fuel assembly.
A Quality Control Inspection Technician reviewed the as-built bundle maps for
fuel assemblies 140148 and 140150 against the load map drawings which were
specified in the computer program header information; however, the technician
did not verify that the load map drawing number listed in the header
information was correct (it was incorrect for fuel assemblies 140148 and
140150).
As a result of the error made in the Robinson fuel assemblies, the fuel vendor
reviewed the as-built lists and records for ROB-13 (Robinson's Cycle 16
refueling) and the remaining assemblies in the Robinson core and approximately
19
1000 additional fuel assemblies the fuel vendor had previously manufactured.
The fuel vendor determined that no other misconfigurations existed. The team
reviewed the documentation of the ROB-13 review and determined that this
method had credibility and that the extent of the problem appeared to apply
only to the six Robinson fuel assemblies in question.
4.2.1 AIT Findings and Conclusions
The team concluded that the control of design information (i.e., the
required location of 'the fuel rods within the fuel assemblies) as it
was translated into the computer system was inadequate; that the level
of responsibility, accountability, supervision, and review associated
with this critical task was inadequate; that the performance of the
quality control overchecks of the computer program information was
inadequate; and that the procedures used to govern the activities were
inadequate.
On November 22, the fuel vendor stated that they had verified 100
percent of the current core load. They compared the as-built
documentation to the core design documents and stated that no other
error was made. the licensee independently verified this
documentation. The team verified that this method had credibility.
The failure of Quality Control to compare as-built information to
design documents was the basic flaw that resulted in the misassembly
of the fuel.
4.3
Assessment of the extent and effectiveness of fuel assembly
verification at the Siemens Power Corporation-Nuclear Division.
The team determined that the fuel vendor used several methods to maintain
accountability of the fuel assemblies and their subcomponents. The methods
included computer-readable bar codes, man-readable serial numbers, hard copy
travellers, and Quality Control overchecks. At the initial stage of fuel rod
manufacture, a serial number was engraved on the lower end cap in the form of
a computer-readable bar code and man-readable number. This serial number was
entered into the Rod Serialization System computer system which tracked the
fuel rod through the assembly process, from manufacture of the lower end caps
through storage of the completed fuel rods.
The lower end cap was welded to the tube of cladding and the lower end cap
serial number was scanned to associate the lower end cap serial number with
the manufacturing order number, the fuel rod group number, the fuel rod part
number, and the clad part number. The fuel rod was further assembled by
loading the fuel pellets, the load spring, out-gassing the fuel rod and
welding the upper end cap to the fuel rod. Following assembly, a leak check,
a uniformity check, a through rod x-ray, and a final inspection for color,
straightness, length, and weld quality was performed. During each step in the
process the lower end cap serial number was computer scanned at the work
station to maintain accountability for completion of the particular portion of
the fuel rod manufacture. At the end of the fabrication process, the fuel
rods were placed in storage bins near the fuel assembly manufacturing area.
20
The Bundle Assembly Data Logger (the computer program) computer system then
tracked the fabrication of the fuel assembly from removal of the fuels rods
from storage through completion of the assembly process.
Fuel assembly manufacturing occurred when a manufacturing order created a
demand and the Quality Control checks had been performed of the fuel assembly
information in the computer program, which had been entered by the clerk, and
Quality Control Engineering had electronically enabled the computer program to
allow the manufacture of a fuel assembly.
Fuel rods were moved from storage onto the order picker by an elevator. The
order picker had 12 trays divisible by 3 to allow for a total of 36 types of
rods to be on the machine at a given time. The computer system was tied to
the order picker which provided a signal light to indicate from which section
rods were to be removed in accordance with the insertion sequence. As the
rods were removed from the order picker, the lower end caps were scanned into
the computer program and the order picker decremented the count of the rods in
the. applicable section.
The fuel rods were moved from the order picker to the loading section of the
insertion table. The fuel rods were scanned as they were placed on the
downward slope of the insertion table in the order of the insertion sequence.
The fuel rods were picked up from the slope of the insertion table by a set of
notched wheels, scanned while in the wheel to verify the insertion sequence,
.and
dumped into a feed trough. The assembler then moved to the proper "x-y"
coordinates, based on the insertion sequence, and the fuel rods were pushed
into the specified "x-y" coordinates of the fuel assembly located on the
assembler table where the fuel rods were then fixed into place.
After completion of the manufacture of the fuel assembly, the computer program
printed out the as-built bundle map, a sequential list of the fuel assembly
"x-y" coordinates and the part numbers and serial numbers of the items which
filled the coordinates (such as fuel rods).
The Quality Control Inspection
Technician compared the as-built bundle map to the load map specified in the
computer program header data to verify that the assembly had been correctly
manufactured.
4.3.1 AIT Findings and Conclusions
The team concluded that the fuel vendor appeared to have an effective program
for assembly verification in most areas. However, the team did conclude that
the fuel vendor performance in the area of entering design information (the
load map fuel assembly pattern) into the computer program computer system and
the subsequent Quality Control overchecks of this process were less than
adequate (See paragraph 4.2 of this report).
In addition, the fuel vendor
indicated that the level of complexity of the fuel assemblies being
manufactured had increased since the manufacturing system had been implemented
in 1984. The team determined that the manufacturing machines and computer
systems involved did not appear to have been worked beyond capacity; however,
.there
did appear to have been an increasing level of complexity of the
information required to be manipulated and verified by the personnel involved
in the manufacturing process.
II
21
4.4
Assessment of the extent and effectiveness of fuel verification at the
site.
Fuel verification at the site consisted of a visual inspection upon receipt.
This included a visual inspection of the assembly externals and the assembly
serial number. This type of inspection is similar to the industry standard.
4.4.1
AIT Conclusion
No method is reasonably available at the site for more detailed verification.
4.5
Assessment of the Siemens Power Corporation-Nuclear Division analysis
of the core neutron flux anomalies.
The team reviewed and evaluated the fuel vendor's performance and response to
the observed core flux anomalies at Robinson during power ascension testing.
The fuel vendor had no role in the Robinson miscalibration of the excore
nuclear power range instrumentation. Their only involvement was to provide
confirming data for the re-calibration procedure, after the incident,
including reasonable weighing factors for edge bundles adjacent to the excore
Nuclear Instruments.
The team assessed the fuel vendor's analysis of the Robinson core power tilt
.and
bundle power anomalies. These were observed at the 30 percent power level
testing, after processing of the in-core flux map measurements, with the
licensee version of the Westinghouse INCORE program using the fuel vendor
provided input files. An interactive licensee and fuel vendor analysis of the
30 percent power measurement results, including rerunning the in-core map, led
to suspension of power ascension until the anomalies could be resolved. The
fuel vendor performed an independent analysis of the in-core measurements
(with their INPAX-W program), confirming that the Cycle-16 core was not
performing as designed. The fuel vendor, after an extensive Quality Assurance
record review, then reported their discovery that six fuel bundles had been
misbuilt, which had resulted in a reload core misconfiguration. The fuel
vendor provided revised INCORE input decks for the as-built condition,
allowing the core power distribution to be evaluated against the design
peaking factors. However, the fuel vendor later discovered an INCORE input
error for all fresh, burnable poison bundle types, after noting an in-core
flux anomaly in other than the misbuilt bundles.
4.5.1
AIT Findings and Conclusions.
The team determined that the fuel vendor's Pressurized Water Reactor
Nuclear Engineering (NE) support, after the observed INCORE anomaly
indication, was appropriate and adequate in confirming the core mis
loading. However, the initial licensee discovery and subsequent fuel
vendor correction of earlier INCORE input file errors should have
triggered a complete deck review by the fuel vendor and could have
alerted both the licensee and the fuel vendor management to potential
problems in the core design process.
22
The team determined that deficiencies in the fuel vendor's analysis
and verification procedures caused the input errors in the in-core
flux mapping computer program (INCORE).
Also,.the fuel vendor's
regeneration of INCORE computer input for the as-built core should
have included a complete deck review, which might have discovered the
computer input errors earlier. The fuel vendor's response after their
INCORE computer input error discovery was appropriate and supplied the
proper level of support.
4.6
Assessment of the licensee's oversight of Siemens Power Corporation
Nuclear Division's fuel analysis and Quality Assurance programs.
The team reviewed selected areas from the fuel vendor reload design analyses
from Robinson cycles-14, -15, and -16 core reloads. The fuel vendor's core
reload design activities began approximately 18 months prior to fuel delivery
upon receipt of the Tentative Scheduled Delivery Date - notification from the
licensee. This includes the projected end-of-cycle performance of the current
cycle, and the estimated energy requirements for the target cycle. Based on
this notification and discussions with the licensee, the fuel vendor's Nuclear
Engineering provided the licensee's Nuclear Fuel Services with a preliminary
fuel bundle and core design. This tentative design, including the number of
bundles and rod types, was also provided to the fuel vendor's Product
Mechanical Engineering for mechanical design and material requirements
development.
Approximately 12 months prior to fuel delivery, the licensee's Nuclear Fuel
Services provided the Final Scheduled Delivery Date - notification to the fuel
vendor, along with the final cycle energy requirement and an upper and lower
energy generation window for the current end-of-cycle. Based on this data,
the fuel vendor provided the final fuel bundle and core design to the licensee
and delivered a Characteristic Specifications document to their Product
Mechanical Engineering for the development of the detailed parts list. The
licensee reviewed the design and provided approval.
At this point, the reload
core design was considered final; however, the fuel vendor documentation had
not undergone Quality Assurance review and approval.
For the Robinson cycle-16 core reload, the original "final" design utilized a
48-bundle split-enrichment batch to achieve a 430 equivalent full power day
operating cycle. At the licensee's request, a further design change was
developed by the licensee and the fuel vendor (November 1992) to allow the 48
bundle reload batch to be reduced to 44 bundles. This was achieved by going
to a low leakage loading core and utilizing a single enrichment bundle design
with more burnable poison fuel types. At this time, the effect of the low
leakage loading pattern to reduce excore detector response was noted by the
licensee.
Based on the changed design, the fuel vendor's Nuclear Engineering revised the
Characteristic Specifications documentation and provided updated load map data
to Product Mechanical Engineering for development of the final parts list and
23
bundle ID core maps. Following their standard Quality Assurance procedure,
the fuel vendor then began the preparation and review process for their formal
documentation packages and provided them to the licensee during the four
months prior to fuel delivery.
The Robinson cycle-16 Reload Batch Design Report, signifying the fuel vendor
Quality Assurance review and approval of the reload design, was issued to the
licensee in April 1993. The Safety Analysis Report was issued in August 1993
and the Startup and Operations Report was issued in September 1993.
The
INCORE monitoring input file (deck) was officially transmitted by letter with
a diskette in October 1993. Several corrections were noted by the licensee
and the fuel vendor provided revisions to the INCORE input file between the
end-of-cycle 15 shutdown and before Cycle 16 startup.
The team's assessment of the licensee activities are as follows:
There was less than adequate licensee oversight of the fuel vendor
cycle-16 core reload design implementation and verification activities
and of the quality assurance procedures concerning the calculation
notebooks.
The licensee conducted independent calculation reviews of the fuel
vendor reload design parameters at their Raleigh offices and primarily
judged design adequacy based on agreement between the two design
models.
The licensee did not conduct onsite audits of the fuel vendor Nuclear
Engineering and Product Mechanical Engineering design activities and
interfaces for the cycle-16 core reload.
The last known Robinson audit that covered neutronic areas was during
the cycle-12 reload design period.
The licensee conducted in-house reviews of the fuel vendor generated
INCORE computer input file; first by processing the file through a
data curve plotting routine, and then by making test runs using
predicted in-core flux measurement data. For the cycle-16 deck, this
process revealed incomplete data and several errors in the initial
fuel vendor transmittal.
Corrective actions did not include a broader
review of the total reload design process.
4.6.1 AIT Findings and Conclusions
The team determined that the licensee's oversight process was inadequate
because the licensee's Nuclear Fuel Section:
failed to review or observe any of the fuel vendor's fuel bundle
assembly manufacturing activities (due to fabrication schedule changes
that occurred during Nuclear Fuel Service's surveillance activities);
and
- I
24
failed to compare the fuel vendor's fuel bundle assembly records to
the characteristic specifications for Robinson's cycle-16 fuel load.
The licensee's Independent Assessment Team investigation of the
Robinson cycle-16 fuel and core reloading problems also concluded that
its oversight of the fuel vendor's fuel manufacturing activities was
less that adequate.
5.0
BROKEN FUEL INSPECTION TOOL DURING REFUELING
5.1
Determination of the Root Cause of the Broken Fuel Inspection Tool
Event.
5.1.1
From the inspection on-site.
On October 11, 1993, licensee personnel discovered that the control rod for
fuel assembly U-24 would not completely insert into the assembly. The control
rod stopped with approximately two feet of full insertion remaining. An
inspection of selected fuel assemblies in the spent fuel pool was in progress
by the licensee's fuel vendor, Siemens Power Corporation, to measure fuel
assembly and fuel rod lengths. These measurements had routinely been made at
other nuclear facilities using similar tools. Further investigation and
conversation between the licensee and the fuel vendor revealed that loose
parts from a damaged fuel inspection tool had been dropped into a control rod
guide tube of assembly U-24.
The team reviewed the fuel vendor site activity log for the conduct of these
measurements and also reviewed the licensee's log of these activities to
determine and verify the sequence of events. Also the vendor's incident
review board report and licensee's self assessment report were reviewed. The
team discussed these activities with licensee oversight personnel and the fuel
vendor team leader who was responsible for the performance of the fuel
measurements. Both the licensee's and the vendor's log books were very brief
and did not provide any detailed information. Neither party had developed any
standards for these logs.
Special Procedure SP-1258, "Fuel Assembly Inspection and Repair," provided
guidance for the conduct of these measurements. This procedure was reviewed
by the team. As part of the procedure guidance for the fuel inspection, the
fuel assembly upper tie plate was removed and a reference plate installed.
This reference plate was held in position by the use of three expandable
anchors which inserted into three guide tube locations. The 'anchors were hand
tightened to secure the plate into position. Although the tool had been used
before, new anchors had recently been fabricated and installed for this
inspection.
From a review of the logs and discussions with personnel, the team developed
the following sequence of events. The fuel vendor personnel measured assembly
S-15H on October 6, 1993.
Following this inspection, the tool (including the
reference plate) was removed from the assembly and placed in a temporary three
foot storage area on the side of the spent fuel pool.
On October 9 at
approximately 9:00 pm, the tool was utilized again on assembly U-24. The fuel
vendor personnel.noted that only two of the expandable anchors engaged and no
25
resistance was noticed when tightening the third anchor. This information was
not included in the fuel vendor site activity log or communicated to licensee
personnel.
The fuel vendor personnel decided the tool could still function in
this condition and continued using the tool.
Following the measurements on
assembly U-24, the tool was used again on assembly U-23.
Fuel assembly
measurements were completed at 7:20 am on October 10, 1993. On October 11,
1993, at approximately 2:00 am, the tool was removed from the spent fuel pool
and the fuel vendor personnel noted damage to the reference plate and missing
parts from the expandable anchor. Specifically, the missing parts included an
expander nut, expander tube, two roll pins, and a portion of the clamp shaft
(see figure X).
Damage to the clamp shaft was also observed which appeared to
be slightly bent. This information was also not recorded in the fuel vendor
site activity log nor communicated to licensee personnel.
Later that same day
at approximately 9:00 pm, licensee personnel experienced difficulty while
inserting the control rod for assembly U-24 at which time the fuel vendor
personnel reported the missing parts from the tool to licensee personnel.
The
licensee performed a visual examination of the control rod and found the
expander tube lodged on the tip of a control rod finger. The licensee
considers the other missing parts to likewise be in the same guide tube where
the expander tube was found based on the construction of the anchor which
would not provide any support for the tube or roll pins once the expander nut
was removed. Licensee personnel examined the guide tube with a plug gage and
found blockage evident near the dash pot region of the guide tube. This also
indicated that the expander nut was still in the guide tube.
Due to the damage noted on the clamp shaft, the fuel vendor believes that the
tool was damaged by an impact from another tool while placed in temporary
storage in the spent fuel pool for the approximate four days between tool
usage. The small, three foot area allocated was congested with many tools
(approximately 13).
Several other tools were moved to this same storage area
during the time frame involved. The licensee's investigation did not reach a
conclusion on how the tool was damaged. Due to the small amount of force
which would be required to overtighten and break the anchor, the team
concluded that the tool could have been damaged either by an impact while in
the storage area or from overtightening.
The team discussed with the vendor the delay in vendor personnel reporting the
missing tool parts to the licensee. Both the licensee and the vendor incident
review board determined that this omission was not a deliberate act but rather
a significant error in judgement by vendor personnel.
Past experience of the
vendor personnel indicated that at other facilities where loose parts were
dropped in the spent fuel pools, licensee personnel had been informed shortly
afterwards. The vendor team had considered that the lost material could be
resolved at a later time and the missing parts were thought to be on the
bottom of the pool and not in a fuel assembly. Although the vendor did not
have a procedure on foreign material exclusion, vendor personnel were trained
by the licensee on the requirements of Procedure PLP-047, Foreign Material
Exclusion Area Program. Access to the foreign material exclusion area was
strictly controlled by licensee personnel and logs were required to document
entering the area or bringing in material.
The vendor was further requested
by the licensee to remove debris from several fuel assemblies prior to fuel
load in the core indicating the sensitivity of this subject. The team
26
concluded that the fuel vendor personnel should have been aware of the
importance that a loose part in the spent fuel pool would have. The
investigative reports also mentioned the poor fitness.for duty of personnel
during this time frame due to illness and being physically tired following
several 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> shifts of work. This poor physical condition could have
contributed to the poor judgement of contractor personnel who failed to
promptly notify licensee personnel of the missing parts/damaged tool.
Due to
these facts, and discussions with the vendor team leader, the team agreed with
the vendor's determination that the failure to promptly report the missing
parts was not a deliberate act.
The licensee performed an engineering evaluation for the continued use of the
fuel assembly with the loose parts inside the guide tube. This evaluation
concluded that this action would be acceptable based upon chemical, thermal,
and mechanical compatibility of the loose parts with the rest of the assembly.
Therefore, licensee personnel installed a thimble plug over the guide tubes
for this assembly. However, as a result of this action, eight fuel assemblies
had to be repositioned in the core loading to substitute a rodded fuel
assembly for the plugged assembly.
5.1.2 Determination of the root cause of the broken fuel inspection tool
event from the fuel vendor Inspection.
Siemens Power Corporation-Nuclear Division Fuel Services department conducted
.fuel
examinations in the spent fuel pool during Robinson's cycle-16 refueling
outage. As part of its fuel examinations, the fuel vendor's site team
measured the length of the assembly and the fuel rods of three fuel assemblies
that were examined in the following order: S-15H, U-24, and U-23. The length
measurements require removal of the upper tie plate from the fuel assembly
followed by the attachment of a reference length plate. The reference length
plate is designed to be attached to the fuel assembly at three guide tube
locations using a remotely activated expandable anchor inserted in each guide
tube. The expandable anchor consist of a clamp shaft with an expander nut
(collet) on each end of a slotted expander tube and two roll pins (one is a
backup) inserted through the clamp shaft below the bottom expander nut. The
expander nuts are drawn into a slotted expander tube by remotely turning the
clamp shaft.
The length measurements of fuel assembly S-15H were completed without incident
on October 6, 1993. On October 10, 1993, during the attachment of the
reference length plate on fuel assembly U-24, the expandable anchor inserted
in guide tube location E-11 failed to tighten within the guide tube. Siemens
Power Corporation-Nuclear Division site team determined that the length
measurements and examinations of fuel assembly U-24 could be completed without
utilizing the malfunctioning expandable anchor. The length measurements of
fuel assembly U-23 were also completed on October 10, 1993, using only two
functioning expandable anchors. On October 11, 1993, as part of its general
pack-up activity, the fuel vendor site team removed the reference length plate
0II
9
27
from the spent fuel pool for the first time since the fuel assembly length
measurement examinations began on October 5, 1993. Once the reference length
plate was removed from the spent fuel pool, the fuel vendor site team
discovered that the malfunctioning expandable anchor clamp shaft was broken,
bent, and missing the following parts:
lower expander nut
expander tube
two roll pins
a portion of the clamp shaft
Upon discovery of the broken expander anchor at Robinson, Siemens Power
Corporation-Nuclear Division Fuel Services Engineering staff destructively
tested an identical expandable anchor and found that the clamp shaft failed in
the region of the uppermost 0.1574-cm (0.062-inch) diameter roll pin with an
applied torsional load of approximately 4.324-N (35-lbin or 0.972-lbf).
In Siemens Power Corporation-Nuclear Division Incident Review Board report
EMF-93-195(P), "H. B. Robinson Fuel Examination, September 21-October 14,
1993, Investigation of Failure to Report Loose Parts," issued on November 5,
1993, the fuel vendor stated, in part, that the issues summarized below were
identified by the Incident Review Board as the root cause/causal factors for
the loose parts event at Robinson:
The applicable Standard Operating Procedure, Fuel Performance, EMF
P71,129, "Fuel Rod and Assembly Length," Revision 0, March 27, 1992,
did not reference the reference length plate Drawing No. ANF-306,200,
"Rod Length Measuring Tool," Revision 0, September 4, 1987.
- .
The fuel vendor's Fuel Services equipment technicians prepared the
reference length plate tool and shipped it to Robinson without
referring to the reference length plate drawing to verify its
configuration. Siemens Power Corporation-Nuclear Division failure to
verify the tool's configuration resulted in placing a tool in service
that was in less than adequate condition because the reference length
plate was designed with four guide pins and it was shipped to Robinson
with only three. The guide pins were designed to aid in orienting the
tool to the fuel assembly by inserting them into the guide tubes. In
addition to its orienting function, each guide pin also served to
protect the expandable anchors.
The fuel vendor concluded that the expandable anchor design was poor
and found no documented engineering review of the expandable anchor
design. According to the fuel vendor,,the design of the expandable
anchor is inconsistent with its design philosophy which dictates that
tool failure will not result in loose parts.
Siemens Power Corporation-Nuclear Division Incident Review Board
concluded that the reference length plate and expandable anchor were
most probably struck from the side and that the impact was likely from
another tool or hardware in the spent fuel pool.
The fuel vendor
added that the difficult handling of heavy tools, crowded conditions,
28
and inadequate preparation of the work area contributed to the
potential for such an impact. Additionally, the fuel vendor noted
that the absence of the forth guide pin reduced the physical
protection of the expandable anchor from external damage.
On November 30, 1993, the NRC's team at the fuel vendor's facility inspected
the reference length plate used at Robinson by the fuel vendor's site team.
The broken expandable anchor clamp shaft and upper expander nut were found in
their original as-built location on the reference length plate.
I
The fuel vendor agreed to perform additional examinations of the fracture
surfaces of the broken clamp shaft and compare its appearance to the fracture
surfaces of the expandable anchor clamp shaft from the destructively tested
expandable anchor. The additional examinations were performed on the two
clamp shafts using both low magnification micrographs taken with an optical
stereo-microscope and scanning electron micrograph mosaics of the fracture
surfaces.
From the results of these examinations, the team determined that the root
cause of the broken expandable anchor was a ductile overload of the clamp
shaft at the upper roll pin location that was induced through multiple
incremental overtorquing events. Siemens Power Corporation-Nuclear Division
analysis of the fracture surfaces and the results of these examinations
(documented in DTP:93:033, "Analysis of Fuel Services Component Failure,"
dated December 3, 1993) reached the same conclusion. The team also determined
-that the location where the expandable anchor overtorquing events occurred
(including the event that resulted in the 2* bend of the clamp shaft) is
indeterminate in that these events may have occurred during functional testing
of the reference length plate in the fuel vendor's mock-up pool, prior to its
shipment to Robinson, or during the fuel assembly examinations performed at
Robinson.
The team also determined that the fuel vendor's original root cause analysis
of the failed expandable anchor (documented in Incident Review Board report
EMF-93-195(P), "H. B. Robinson Fuel Examination, September 21-October 14,
1993, Investigation of Failure to Report Loose Parts," issued on November 5,
1993) was less than adequate because it did not determine the mechanical
failure mechanism of the broken clamp shaft.
5.1.3 AIT Findings and Conclusions
During fuel preparation for core load, loose parts had been found in a fuel
assembly as a result of a fuel inspection reference tool breaking. The fuel
vendor had been conducting fuel inspections in the Robinson spent fuel pit.
The root cause of the broken fuel inspection tool was attributed to the fuel
vendor design control problems and inadequate licensee oversight. The current
fuel vendor design control system had been set up in 1988 with no requirement
to use this methodology on items constructed after this date but built to an
.earlier
design. This particular design was completed in 1987 and the tool
constructed in 1992 and refitted in 1993.
29
The licensee's analysis of the small, unrecovered parts found that they were
confined to the fuel assembly guide tube and presented no future threat to
fuel integrity. The team agreed with this finding.
5.2
Effectiveness of Licensee Oversight of Contractor Fuel Handling
Activities
The team reviewed the involvement of licensee personnel during the fuel
measurements performed by the fuel vendor. This item was discussed with the
licensee and contractor personnel involved. Also the team reviewed the
licensee's assessment report..
According to licensee plant personnel, oversight of this activity included 24
hour coverage using 2 12-hour shift rotations. Licensee personnel were
present on the fuel handling floor during the fuel assembly inspections. Only
verbal guidance was provided to licensee personnel on their responsibilities
for oversight functions which consisted of assuring that vendor personnel
adhered to the licensee's procedures.
The team discussed this coverage with licensee personnel and was informed that
in some cases, concurrent activities occurred in the pool. This interfered
with direct oversight of the vendor activities. This occurred during the tool
removal, when licensee personnel were concurrently inspecting fuel assemblies
.for
debris. Licensee involvement in observing the video display for the
debris inspection prevented the direct oversight of contractor personnel when
the tool was removed.
The licensee's independent assessment report of this event concluded that
licensee personnel were not always present during the fuel measurements. This
contributed to the poor communication between the licensee and the vendor.
The team could not determine from the logs how long licensee personnel were
actually present for the fuel inspections.
Contractor activities for this fuel inspection were controlled by licensee
Procedure SP-1258. This procedure included attachments with the vendor's
procedure for the ultrasonic inspection, repair, and examination of fuel
assemblies (EMF-1576). The completed procedure was reviewed by the team as
well as the licensee's safety evaluation package of the procedure. The team
noted that the attached vendor procedure contained references to detailed
vendor procedures for the performance of the fuel rod and assembly length
measurement (EMF-P71,129) and for the upper tie plate removal/reinstallation
(ANF-P71,032). These two procedures were utilized by the vendor and involved
partial disassembly of the fuel assemblies for the inspections including tie
plate removal and the removal of fuel rods from the assembly. The team noted
that these quality activities were not encompassed by the licensee's safety
review of Procedure SP-1258.
The licensee provided training for the contract personnel on the implementing
procedure and requirements of Procedure PLP-037. Procedure SP-1258, section
O6.2,
contained a specific precaution on the requirements for foreign material
exclusion areas. The training consisted of handing out the procedures and
allowing contractor personnel to "self study" the handouts. The contractor
30
personnel acknowledged by signature the accomplishment of this training. The
licensee's assessment report concluded that the contractor indoctrination for
foreign material exclusion requirements was not effective.
The licensee's self-assessment report also identified that planning and
coordination for this activity was inadequate based on insufficient space
availability and availability of support services. This report also stated
- that the responsibilities of licensee personnel who provided the oversight
function were not clearly defined.
The team noted from log book entries that both contractor and licensee
personnel were physically tired and in some cases ill.
This condition was not
noted in subsequent licensee investigations.
5.2.1
AIT Findings and Conclusions
The team concluded that the licensee oversight of contractor activities was
considered to be less than adequate; this was verified by interviews with
personnel and review of logs. The contributing causes were:
Poor planning and coordination of fuel inspection.
Failure to identify and adjust staffing of personnel when conditions
changed.
Lack of clearly defined responsibilities.
Poor safety review of vendor procedures for fuel inspection.
A review of the licensee's assessment report revealed their causes of the
event to be:
the poor planning and coordination of the fuel inspection
activities, lack of licensee management to identify and adjust staffing of
personnel when degradation of physical conditions was indicated, lack of
clearly defined responsibilities for licensee personnel overseeing this
activity, and a poor safety review of procedures utilized by the contractor in
performing these inspections.
5.3
Assess the adequacy of Siemens Power Corporation-Nuclear Division
Quality Assurance program for the manufacture of special fuel tools.
The reference length plate used at Robinson was designed in September 1987 and
has been used several times at Robinson and Tihange in Belgium (both plants
have 15 x 15 Pressurized Water Reactor fuel assemblies).
However, the
reference length plate design depicted on Drawing No. ANF-306,200, Revision 0,
had not been reviewed in accordance with the fuel vendor's design review and
design control measures established by the Quality Assurance program in 1988.
Siemens Power Corporation-Nuclear Division failure to evaluate the reference
length plate design in accordance with its Quality Assurance program is
considered by the team to be a significant contributing factor in the loose
parts event at Robinson.
In November 1990, Siemens Power Corporation-Nuclear Division Fuel Services
Engineering apparently recognized the poor design of the expandable anchor
that failed at Robinson. Drawing No. ANF-306,200, Revision 1, approved on
31
November 14, 1990, revised the expandable anchor design by (a) increasing the
diameter of the clamp shaft, (b) threading the end of the clamp shaft and
providing a threaded locking sleeve to retain the lower expander nut, and
(c) inserting a roll pin through the threaded locking sleeve and shaft to
ensure the locking sleeve would not loosen and back-off. Implementation of
these design changes would have prevented the loose parts event at Robinson.
However, the revised expandable anchor design was not manufactured or ever
utilized by the fuel vendor. Siemens Power Corporation-Nuclear Division's
failure to incorporate the enhanced expandable anchor design in the reference
length plate was a missed opportunity that contributed to the loose parts
event at Robinson.
Moreover, during its preparation of the reference length plate for shipment to
Robinson, the fuel vendor missed another opportunity to prevent the loose
parts event at Robinson. In September 1993, when the Fuel Services personnel
retrieved the reference length plate from storage in preparation for its
shipment to Robinson, it was discovered that one of the expandable anchors had
missing parts (the lower expander nut, expander tube, and the roll pins).
The
fuel vendor responded to the discovery of missing parts by fabricating three
new expandable anchors to the old, 1987 design. The fuel vendor failed to (a)
determine what had happened to the missing parts (i.e., were the parts lost
during the reference length plate's last usage, which was at Robinson during
its cycle-15 refueling outage, and if so, can the missing parts be located),
(b)
fabricate the replacement expandable anchors in accordance with the
revised 1990 design, and (c) perform an engineering evaluation of the newly
fabricated expandable anchors' design in accordance with the requirements of
the Quality Assurance program.
Subsequent to the team's identification of Siemens Power Corporation-Nuclear
Division's less than adequate actions regarding the missing parts, the
licensee at Robinson investigated the potential that the missing parts may
have entered Robinson's spent fuel pool or core during Robinson's cycle-15
refueling outage. Although it was not possible to establish with certainty
what happened to the missing parts, the licensee, on the basis of all
available information, determined that the missing parts were not located in
the Robinson spent fuel pool or core. The team reviewed the licensee's
evaluation and accepted its conclusion.
5.3.1 AIT Findings and Conclusions
From its review of Siemens Power Corporation-Nuclear Division's Quality
Assurance program, the team determined that for those fuel tools developed
since the implementation of the Quality Assurance program, the Quality
Assurance program appeared to be adequate. Siemens Power Corporation-Nuclear
Division's Quality Assurance program for the manufacture of special fuel tools
began in 1988 when the Fuel Services department first implemented Siemens
Power Corporation-Nuclear Division's Quality Assurance Manual EMF-1.
However, for those fuel handling tools that were developed by the Fuel
.Services
department before the implementation of Siemens Power Corporation
Nuclear Division's Quality Assurance program, Siemens Power Corporation-
32
Nuclear Division efforts to incorporate those tools into the Quality Assurance
program's design review and design control measures appear to be less than
adequate; as demonstrated by the reference length plate (Drawing No. ANF
306,200, "Rod Length Measuring Tool," Revision 0, September 4, 1987),
described above.
5.4
Assessment of the effectiveness of Siemens Power Corporation-Nuclear
Division's program for notifying licensees of known deficiencies in either
hardware or services provided.
The team reviewed Siemens Power Corporation-Nuclear Division's program for
review and notification to customers of identified deficiencies in hardware or
services, with particularly interest in the three deficiencies identified
during the Robinson event. The identified deficiencies were, (a) the presence
of loose parts in the fuel pool, (b) the incorrect manufacturing of six fuel
assemblies, and (c) the potential error in the generation of transport
correction factors used in the generation of the INCORE code.
Siemens Power Corporation-Nuclear Division's program is controlled by Policy
Guide 10.2, "Nuclear Safety Hazards Reporting," dated December 17, 1991.
Policy Guide 10.2 was previously reviewed during a February 1992 NRC
inspection at Siemens Power Corporation-Nuclear Division (see Inspection
Report 99900081/92-01). During the 1992 inspection, the team found the Policy
OGuide contained all necessary requirements of 10 CFR Part 21.
However, the
1992 inspection did identify an area of concern regarding the language
relating to who has responsibility for performing evaluations. This concern
was again raised to Siemens Power Corporation-Nuclear Division's management
during this inspection.
As previously discussed, the fuel vendor Fuel Services personnel failed to
report the presence of loose parts in the spent fuel pool to Robinson's staff
in a timely manner. The loose parts resulted from a failure of the reference
length plate expandable anchor. The licensee was not notified of the failure
until Robinson personnel were unsuccessful in loading a control rod into fuel
assembly U-24, approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> after the fuel vendor site team first
identified the parts as missing. The fuel vendor convened an Incident Review
Board to investigate the circumstances surrounding the event and to determine
10 CFR Part 21 applicability. Incident Review Board report EMF-93-195(P),
"Incident Review Board Report, H. B. Robinson Fuel Examination, September 21
October 14, 1993, Investigation of Failure to Report Loose Parts," concluded
that there were no 10 CFR Part 21 implications from this event. The Incident
Review Board report based its conclusion on justifying the use of assembly
U-24 using a thimble plug device and moving its location in the core. The
team questioned the adequacy of this conclusion in light of potential generic
considerations regarding the use of the fuel tool at other facilities and the
potential for loose parts escaping the guide tube and damaging other
assemblies in the core. The fuel vendor provided the team with engineering
evaluation, "H. B. Robinson - Project Variance Evaluation of a Piece of a Tool
End in a Guide Tube of Assembly U-24," which was provided to the licensee in
an October 13, 1993, letter (RAC:93:165). The engineering evaluation provided
sufficient evidence that the loose parts in assembly U-24's guide tube (E-11)
33
could not escape from the tube. The fuel vendor also informed the team that
the tool was only used at Robinson and one foreign plant (Tihange in Belgium).
Since Robinson is the only U.S. commercial facility that has used the tool,
and they were notified approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> after the fuel vendor site team
identified the missing parts, further notification by the fuel vendor was
unnecessary.
The problem with the incorrect manufacturing of six fuel assemblies was
identified by the licensee during power distribution mapping at Robinson at
approximately 30 percent power on November 18, 1993. The fuel vendor was
notified of the power distribution anomalies and convened a Hazards Review
Board the same day. The board concluded that the manufacturing error
constituted a defect under 10 CFR Part 21 which could potentially pose a
significant safety hazard and recommended that the condition be reported to
the NRC. The fuel vendor further recommended that the licensee make the
initial notification to the NRC. The licensee fulfilled the notification
requirement later on November 18, 1993. Both Siemens Power Corporation
Nuclear Division and the licensee's actions regarding 10 CFR Part 21 were
appropriate and in accordance with regulatory requirements.
The team further examined Siemens Power Corporation-Nuclear Division actions
associated with the incorrect manufacturing of the fuel assemblies with regard
to the identification of potential generic aspects. The fuel vendor formed an
Incident Review Board to investigate this matter on November 19, 1993. The
team reviewed draft Incident Review Board report EMF-93-209(P), "Incident
Review Board Report, Misconfigured Fuel Assemblies at Robinson," dated
December 1993. The report includes a review of reload records for Robinson,
Susquehanna, Dresden, Grand Gulf, Peach Bottom, Kuosheng 2, WNP-2,
Comanche Peak, and Laguna Verde to assure that each rod was located in the
correct position in each assembly. The review encompassed approximately 1240
fuel assemblies. The fuel vendor did not identify any other mispositioned
rods.
The team reviewed an internal memorandum of November 28, 1993, Siemens Power
Corporation, subject "Potential Error in Generation of the Transport
Correction Factors Used to Generate INCORE Analytic Factors" (KCS:93:016).
The anomalies were identified during an examination of the INCORE 30 percent
power maps for Robinson. In accordance with ANF-POO,002, Quality Assurance
Procedure No. 3, "Design Control for Nuclear Fuel," the fuel vendor began an
assessment of the potential error's impact on the Robinson cycle-16 fuel load.
The assessment is scheduled to be completed by December 15, 1993. The team
discussed the potential generic implications of the error with the fuel
vendor. The fuel vendor indicated that the INCORE Computer Code was a
Westinghouse Code and had only been used by the fuel vendor for Robinson and
that Robinson had been informed of the potential error (the fuel vendor did
indicate that they intended to use the code for a future reload of Shearon
Harris).
The team concluded that 10 CFR Part 21 requirements had been met.
The team also discussed the potential for the error being applicable to other
facilities which use the INCORE Computer Code. The fuel vendor stated that
the error had been made by the fuel vendor personnel and was not inherent in
the code. Since the fuel vendor evaluation of this issue was not complete,
this item should be examined during a future inspection.
.34
5.4.1
AIT Conclusion
The team concluded that 10 CFR Part 21 requirements had been met.
6.0
ASSESSMENT OF LICENSEE INVESTIGATION OF THESE EVENTS
6.1
Assess the effectiveness and thoroughness of the licensee's
investigation of these issues.
The team reviewed their independent evaluation of the events and root causes
against the licensee and the fuel vendor findings. The team concluded that
the licensee and its fuel supplier have done a thorough job of review and
their root cause determinations are reasonable. The licensee's and the fuel
vendor's level of management involvement in their investigations and in their
internal critiques of their investigations has been indepth and involved the
highest levels of their respective organizations. The AIT findings basically
agree with that of the licensee as noted below and in specific places in the
report.
6.1.1
AIT Review of the licensee Nuclear Instrumentation Miscalibration
Review Team Assessment
The licensee's investigation attributed the root cause for the nuclear
instrumentation miscalibration event to be the inadequate implementation of
.corrective
action following similar industry events. A casual factor for the
event was determined to be an improper methodology used in calculating the
power range currents.
Their investigation found that the licensee calculated a correction factor to
apply to the previous cycle's 100 percent Power Range Nuclear Instrumentation
currents by multiplying the previous current by a ratio of previous cycle
average relative power for three fuel bundles to predicted average relative
power for fuel bundles in the new core load. In this calculation, the two
nearest, outer diagonal fuel assemblies and a third inner assembly (second
diagonal) were used for relative power comparisons. This methodology differed
from that recommended by the Nuclear Steam Supply System vendor Westinghouse
(see figure Z).
This licensee assessment was issued on December 3, 1993. The members and
their scope are outlined in Appendix C.
The AIT agreed with these findings and conclusions and in addition, found that
the Nuclear Instrumentation miscalibration was the result of an incomplete
understanding of the core geometry considerations by the procedure writer and
inadequate review by the corporate fuels group.
6.1.2. AIT Review of Licensee's Robinson Fuel Loading Investigation Team
Assessment
The licensee team considered the fuel vendor errors and their not being
detected by the licensee as Principal Causes.
35
The licensee's investigation attributed contributing causes for the
failure of a fuel inspection tool and consequent loose parts event to
be:
a. inadequate tool design
b. inadequately defined roles and responsibilities
c. failure to follow proper foreign material exclusion practices.
The AIT agreed with the above findings and conclusions with the addition of
the contribution of the poor physical condition of contractor and licensee
personnel.
The licensee's investigation attributed the contributing causes of the
misconstructed fuel bundles to be:
a. the licensee's failure to ensure fabricated fuel meets design
requirements because of a lack of management direction and
inadequacies in review and evaluation programs; and
b. the fuel supplier's fabrication of bundles with incorrect
Gadolinium rod placement, caused by inadequacies in procedures,
accountability, training and overchecks.
The AIT agreed with the above findings and conclusions.
The licensee's investigation attributed the contributing causes of the
core design data problem to be:
a. Fuel supplier errors in producing the input due to the work being
hurriedly done, inadequate procedures and inadequate data checkout
tools.
b. Inadequate licensee oversight and review of the supplier analyses.
The AIT agreed with the above findings and conclusions.
This assessment was issued on December 8, 1993. The membership and their
scope are outlined in Appendix C.
7.0
EXIT MEETING.
On December 6, 1993, the team, accompanied by the Deputy Regional
Administrator for Region II, conducted a public exit meeting at the Robinson
site. The licensee and NRC personnel attending this meeting are listed in
Appendix D. Proprietary material has not been included in this inspection
report. During the exit, the team summarized the scope and findings of the
inspection. There were no dissenting comments from the licensee of the
findings.
APPENDIX A
H. B. ROBINSON AUGMENTED INSPECTION TEAM (AIT) CHARTER
A.
Basis
On November 16, 1993, during startup of H.B.Robinson Unit 2, reactor core flux
anomalies were identified during flux mapping at approximately 30 percent power.
A second flux map confirmed core design problems. Other problems were identified
during the startup including the Power Range Nuclear Instruments reading 10
percent below the actual power level of 30 percent.
B.
Scope
1. Develop and validate the sequence of events associated with the
November 12, 1993, startup until Hot Shutdown was reached on
November 17, 1993.
2. Assess the root cause and safety significance of the core neutron flux
anomalies with regard to fuel and technical specification limits.
3. Determine the root cause of the miscalibrated nuclear instruments identified
during startup.
4. Assess operator performance relative to the nuclear instrumentation
miscalibration problem.
5. Assess the adequacy of station nuclear instrumentation calibration and
refueling procedures.
6. Determine the root cause of the broken fuel handling tool event, and the
effectiveness of licensee oversight of contractor fuel handling activities.
7. Assess the effectiveness and thoroughness of the licensee's investigation of
these issues.
8. Assess the cause and extent of the fuel manufacturing errors at Siemens Fuel
Manufacturing Facility and the extent and effectiveness of fuel verification
at the site.
9. Assess the adequacy of the licensee's oversight of Siemens' fuel analysis and
Quality Assurance programs.
10. Prepare a special inspection report documenting the results of the above
activities within 30 days of the inspection completion.
C.
Team Members
Team members will include:
Team Leader, Senior Resident Inspector, Robinson
Resident Inspector, Reactor Physics Specialist, License Examiner, and
Vendor/Quality Assurance Inspectors to inspect at the Robinson Site; follow-up at
the Siemens Fuel Manufacturing Facility will be conducted by the Reactor Physics.
Specialist and the Vendor/Quality Assurance inspectors.
Appendix A
2
SUPPLEMENT TO AUGMENTED INSPECTION TEAM (AIT) CHARTER FOR H. B. ROBINSON AND SIEMENS FUEL
MANUFACTURING FACILITY
A.
Basis
On November 16, 1993, during startup of H. B. Robinson Unit 2, reactor core flux
anomalies were identified during flux mapping at approximately 30 percent power. A
second flux map confirmed core design problems.
Other problems were identified which
included the Power Range Nuclear Instruments reading 10 percent below the actual power
level of 30 percent and a broken fuel handling tool.
B.
Scope
1. Determine the root cause of the broken fuel handling tool event.
2. Assess the adequacy of Siemens' Quality Assurance program for the manufacture of
special fuel tools.
3. Assess the cause and extent of the fuel manufacturing errors at Siemens Fuel
Manufacturing Facility and the extent and effectiveness of fuel
assembly
verification at Siemens.
4. Assess the adequacy of the licensee's oversight of Siemens' fuel analysis and
Siemens' Quality Assurance programs.
5. Assess the Siemens analysis of the core neutron flux anomalies.
6. Assess the effectiveness of Siemens' program for notifying licensees of known
deficiencies in either hardware or services provided.
7. Prior to exiting the Siemens'
facility brief the AIT team leader of the
preliminary inspection findings via telephone.
8. Provide inspection results in writing to AIT team leader within one week of
exiting the Siemens' facility.
C.
Team Members
Team members will include:
Reactor Physics Specialist -
Edward D. Kendrick and
Vendor/Quality Assurance Inspectors -
Steven Matthews and W. H. Rogers.
APPENDIX B
AIT REVIEW TEAM
Corrective Action Assessment Team
Warren Dorman
Team Leader, RNP CAP/0EF
Franklin Murray
HPES RNP
BND
SCOPE:
Review past RBN performance (NAD,
INPO, NRC,
and other assessments) and evaluate
effectiveness of RBN CAP program in correcting previously identified performance
issues and predicting areas requiring additional attention.
Robinson Fuel Loading Investigation Team
Lou Martin
Team Leader
Bob Toth
INPO, Assistant Team Leader
Dave Waters
(Misconfiguration Focus)
John Eads
(Inspection Tool Failure Focus)
Jim Thompson
(Power Escalation Recommendation Focus)
tside Member
(Assist With Fuel Fabrication Focus)
OPE:
(1) Conduct a detailed root cause analysis of the core loading problems
of the following:
-
The Siemens inspection tool failure and the resulting fuel assembly
relocation.
-
The misconfiguration of fuel assemblies.
-
The adequacy of the licensee's oversight of Siemens activities both
onsite refueling and the core analysis activities.
(2) Review documentation and interview personnel at HBR,
Fuels,
and
Siemens to determine root cause and why not identified by the
licensee and Siemens prior to fuel load.
(3) Evaluate casual factors for other impact.
(4) Complete tasks prior to head replacement.
pendix B
2
Nuclear Fuels Instrumentation Investigation Team
C. S. Hinnant
Team Leader
Chip Moon
Operations
Bryan Waldsmith
Operations
Danny LaBelle
Fuels
Jo Ellen Westmoreland
Reactor Engineer
Franklin Murray
HPES
Dick Cady
CAP/Maintenance
Brian O'Donnell
David Coates
Training
SCOPE:
Conduct investigations following the following guidelines:
(1) Operations:
time line; events and casual factor;
ERFIS data;
Operations logs; plant data; -and power ascension coordination
(2) Fuels:
fuel vendor data; corporate fuel design data; fuels/plant
interface.
(3) Reactor engineering interface with fuels and operations and
comparison of Harris lessons learned with HBR corrective actions.
(4) HPES:
Event analysis using "yellow sticky" method and HBR actions
from similar O.E. identified events.
(5) CAP/Maintenance:
PL-026 and format of/process to develop final
report.
(6) INPO:
Industry experiences.
(7) Training:
Reactivity management training, and startup from RFO
training.
APPENDIX C
ATTENDANCE LIST AT EXIT DECEMBER 6, 1993
S. A. Bilings
Regulatory Affairs
T. A. Peebles
AIT Leader, RIH
L. A. Reyes
Deputy Regional Administrator, RIH
E. D. Kendrick
Nuclear Engineer, NRR
C. R. Ogle
AIT Member, RI
B. H. Rogers
Reactor Engineer/UIB, NRR
S. M. Matthews
Quality Assurance Engineer DRIL/VIB, NRR
D. Waters
Manager, Regulatory Affairs, CPO
S. Zimmerman
Manager, Nuclear Fuel Management & Safety Analysis
M. Pearson
Plant General Manager, Robinson
C. R. Dietz
H. W. Habermeyer, Jr.
W. S. Orser
Exec VP, Nuclear Generation
R. E. Rogan
Manager, CP&L, Licensing
B. H. Clark
Manager, Maintenance
A. R. Wallace
Manager, Licensing/Regulatory Programs
M. Herrell
Manager, Training
T. P. Cleary
Manager, Technical Support
J. Guibert
Consultant to CP&L
D. G. McAlees
Sr. VP & GM, Siemens, Nuclear Division
N. Morgan
VP Engineering, SPC-ND
Watts
Electrical Dept-SCPSC
S. Stancil
Nuclear Bus. Oper., CP&L
S. Singh Bajwa
NRC, NRR, Projects
B. L. Mozafari
NRC, NRR, PDII-1
P. J. Jordan
Manager, Nuclear Human Resources, CP&L
W. S. Baum
Nuclear Employee Relation, CP&L
G. Newsome
Nuclear Engineer, CP&L
W. Pridgen
CP&L, Manager
C. S. Olexik
Manager, Plant Assessment, CP&L
K. Clark
Public Affairs, NRC, RII
H.B. ROBINSON CORE
AS-rDESIGNED
AS -BUIL TN
03
03
05
06
06
07
07.I
Itlot
112
$
0V
14
14.
15
1
P H M LK
J
C
F E 0C
A
R PN M L KJ
0A
Q UAIDRA
cowr..ii
GDOMJA RODS
4 000-
-2
50
0
a IE
E Fl'.E
-'?I)
I.IC 2A
-01-11C-2
50 0 "D
L I
HA
-
LEF T HAND/
CL
AMPSHAFT0:125
06 D1A0.
EXPANDER
TU~fE-j
I
LENGTH PLATE-L..I
IREF)~ral
Cacat@
_____________
____
Il~
LEGT
MEASURIN
TOO
(H ROBISON
FIGURE Z
RECOMMENDED ASSEMBLY TO USE FOR NI CALIBRATION
Igo
180*
Rf PNM
LK
J
S
F
EDC
A
.3
6
7
90*--8
4300-.
-2-70
A
CCC
A
Thre Loop Plant