IR 05000315/1989025
| ML17328A168 | |
| Person / Time | |
|---|---|
| Site: | Cook  |
| Issue date: | 09/13/1989 |
| From: | Axelson W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML17328A167 | List: |
| References | |
| 50-315-89-25, 50-316-89-25, NUDOCS 8909290056 | |
| Download: ML17328A168 (32) | |
Text
U ~ S.
NUCLEAR REGULATORY COMMISSION
REGION III
Reports No. 50-315/89025(DRP);
50-316/89025(DRP)
Docket Nos.
50-315; 50-316 Licenses No. DPR-58; DPR-74 Licensee:
American Electric Power Service Corporation Indiana Michigan Power Company 1 Riverside Plaza Columbus, OH 43216 Facility Name:
Donald C.
Cook Nuclear Power Plant, Units 1 and
Inspection At:
Donald C.
Cook Site, Bridgman, Michigan Inspection Conducted:
August 15 through 18, 1989 AIT Members:
B. Burgess, Team Leader I. Ahmed J. Giitter B. Jorgensen R. Westberg D. Butler D. Passehl l!
Approved By:
W. "L. A elson, Chief Reactor Projects Branch
~
/S S~/
Dat Ins ection Summar Ins ection on Au ust 15 throu h
1989 Re orts No. 50-315/89025 DRP No. 50-316/89025 DRP A
I
- i (AT)
the circumstances surrounding the Unit 2 reactor trip of August 14, 1989, including:
sequence of events; root cause(s)
of equipment failures; adequacy of maintenance and surveillance on failed equipment; adequacy of procedures; adequacy of operator responses; adequacy of licensee reporting; and adequacy of licensee event followup and review.
Results:
No violations or deviations were identified in any of the areas inspected.
No significant operational safety parameters were approached or exceeded.
Equipment problems with one exception (Steam Generator wide range level indication), were not the result of design inadequacies.
Maintenance and surveillance practices were not found to be materially deficient.
Operators performed correct actions in responding to the event; these actions conformed to procedures or were conservative.
Licensee reporting and overall event followup were adequate.
8909290056 890%9'DR ADOCK 050003l5 G
PNU
AUGMENTED INSPECTION TEAM REPORT U. S.
NUCLEAR REGULATORY COMMISSION
REGION III
D.
C.
COOK UNIT 2 REACTOR TRIP/DEGRADED CLASS IE POWER AUGUST 14, 1989 Inspection Reports No.
50-315/89025.
50-316/89025
TABLE OF CONTENTS l.
Event Summary
~
~
~
~
~
~
~
~
~
~
~
~
~
2.
Persons Contacted.......
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
3.
Sequence of Events
4.
Equipment Failures Causes and Consequences..
~
~
5.
Maintenance and Surveillance Practices
~
~
~
~
~
~
~
~
~
~
~
~
1 1 6.
Procedures
7.
Operator Responses.
~
~
~
~
~
~
~
~
8.
Licensee Management Response and Reporting
............
9.
Licensee Event Followup and Review.
Attachments:
1.
AIT Charter 2.
Licensee Commitment Letter 3.
Event Sequence Time Line 4.
Equipment Affected by CRID-IV 5.
Procedures Reviewed
Details Event Summar The Unit 2 reactor tripped from 100 percent power at 4:01 p.m.
(EDT) on August 14, 1989, as a result of a problem in a transfer switch supply to control room instruments.
The trip occurred when operators restored the normal (Class IE) power supply to Control Room Instrument Distribution (CRID) panel IV, which had automatically transferred to its alternate (non-Class IE) power supply some 20 minutes earlier.
Upon transfer of CRID-IV to normal power, the CRID panel experienced severe undervoltage leading to the reactor trip and the loss of numerous instrument indications.
Among the instruments and controls lost were all four wide-range steam generator level indicators, condenser steam.dumps, Solid State Protection'ystem (SSPS)
Train B output relays, various No.
reactor coolant pump instruments, and No.
3 and No.
4 safety injection accumulator monitor ing functions, and others.
A more complete list is attached to this report (attachment 4).
Immediately after the reactor trip, three control rod "rod bottom" lights failed to illuminate.
One of these did illuminate a short time later.
The turbine stop valve status lights did not indicate the turbine had tripped.
Excessive heat from overcurrent to unfused items resulted in a
"smoke smell" in the control room, and the control room smoke detection circuit alarmed.
Substantial relay "chatter" was audible in the control room for a considerable time after the trip.
When the turbine (which had tripped)
slowed down sufficiently, the turning gear motor failed and reports of smoke and damage were received in the control room.
Critical plant parameters were maintained within normal post-trip ranges during the transient recovery, with some systems requiring manual operator control.
CRID-IV was transferred to another alternate power supply after about 45 minutes, and restoration of individual instruments and controls was substantially complete within 90 minutes of the reactor trip.
NRC Region III personnel were briefed by the Senior Resident Inspector, who had followed post-trip recovery in the control room, about two hour s after the event.
A Region III instrumentation specialist was dispatched to aid in technical evaluation of the event, and to recommend further escalation as necessary.
The licensee notified the NRC Operations Center pursuant to 10CFR50.72 within the specified four hour limit.
Estimated time to repair failed equipment required that the reactor be cooled down to MODE 5 pursuant to applicable Technical Specifications.
On August 15, 1989, following consultations among NRC personnel onsite, in Region III (Chicago),
and in the Office of Nuclear Reactor Regulation (NRR),
an NRC Augmented Inspection Team (AIT) was established to review the event.
The AIT, headed by a Region III Section Chief, incorporated the NRC personnel already onsite and added personnel from NRR.
This report describes the AIT activities and findings.
The Team charter is included as Attachment No. On August 16, 1989, the licensee submitted a letter to NRC Region III (Attachment No. 2) covering actions taken and planned to be completed prior to unit restart from the reactor trip.
The AIT independently verified licensee performance of these actions.
Persons Contacted M. Alexich, Vice President Nuclear Operations
- W. Smith, Jr., Plant Manager A. Blind, Assistant Plant Manager Administration
- J. Rutkowski, Assistant Plant Manager - Production S.
Brewer, Manager, Nuclear Safety and Licensing K. Munson, Senior Engineer - Instrumentation, AEPSC
"B. Svennson, Licensing Activity Coordinator
"K. Baker, Operations Superintendent J.
Sampson, Safety and Assessment Superintendent
"T. Beilman, IKC Department Superintendent
- J. Droste, Maintenance Superintendent T. Postlewait, Technical Superintendent Engineering
"Denotes personnel attending the exit interview on August 17, 1989 The AIT also contacted a number of other licensee and contract employees and interviewed operations, maintenance, and.technical personnel.
Se uence of Events The event began at about 3:40 p.m.
(EDT) on August 14, 1989 with the initial abnormal indication being a control power failure on power range nuclear instrument channel N-44.
A blown fuse was immediately suspected and quickly confirmed as the causal factor for N-44 failure.
The Instrument and Control (IKC) group was summoned to investigate and the channel was declared inoperable.-
Coincident with the N-44 fai lure, Control Room Instrument Distribution panel/bus No.
4 (CRID-IV) auto-transferred to its alternate (non-Class IE)
power supply.
This was initially diagnosed as a probable consequence of the N-44 failure, but ultimately proved more likely to be the cause, as discussed elsewhere in this report.
CRID-IV was also declared inoperable as required by Technical Specifications.
Effective at 3:40 p.m., Unit 2 was in two Technical Specification Limiting Conditions for Operation.
The first required tripping bistables associated with N-44 within one hour,'he second limited continued operation with GRID-IV inoperable to a maximum of eight hours.
The bistables were tripped at 3:45 p.m. in compliance with the first requirement.
Inspection of the N-44 channel and CRID-IV indicated the Class IE CRID power supply appeared normal, so the decision was made to restore CRID-IV to OPERABLE before proceeding with troubleshooting N-44.
The remaining N-44 channel fuses were pulled to assure it was isolated.
The Class IE bus frequency was slightly low (59.5 vs 60 Hz)
so it was adjusted by an
'8C supervisor to synchronize the circuitry for the transfe Operation pr'ocedure
- "2-OHP 4021.082.008,
"Operation of GRID Power Supplies,"
Section 6.3 covers switching from alternate source to inverter.
It includes a precaution to the effect "...
a thorough investigation should be made before switching back
.
.
.."
Subsequent events implicated the thoroughness of the investigation (which concluded N-44 was the source of the problem) but a more detailed evaluation concluded the actual fault was not reasonably detectable, as discussed in Section 4 of this report.
Following Section 6.3 of the procedure, the various available indications were confirmed proper for the transfer.
When the transfer pushbutton was depressed at 4:01 p.m., the reactor tripped, a number of instruments were lost, considerable relay "chatter" was evident in the main control room, and a "smell of smoke" was.detected there.
The control room fire detection circuit also 'alarmed, among many alarms actuated.
Three control rod "rod bottom" lights did not illuminate, nor did turbine stop valve status lights.
The Augmented Inspection Team (AIT) reviewed instrument strip-charts, the Operations Sequence Monitor (OSM) output, the Prodac-250
"Trend" and
"Alarm" outputs, and the control room logs.
The oscillogram and turbine events monitor traces were reviewed.
Licensee employee
"personnel events summaries" and followup procedure records were also examined.
All operations and 18C personnel directly involved in the event were interviewed.
The purpose of this effort by the AIT was to develop and validate the sequence of events for determination of the adequacy of equipment and operator response.
No significant abnormalities were identified in either equipment or operator response.
An annotated, detailed sequence of events is included as Attachment No.
3 to this report.
E ui ment Failures Control Room Instrumentation Distribution CRID No.
4 Bus (1)
~Back round The 120 Vac vital instrument bus system consists of four separate buses which are supplied by four independent 7.5 kVA, single phase static inverters.
Two of the inverters connect to one of the station batteries, the other two connect to a second station battery.
The static inverter consists of a power switching circuit (inverter) which converts a 250 Ydc input ( lE source) to a regulated 120 Vac sinusoidal output.
The output of the power switching circuit is applied through a
static switch (part of the static inverter) which electrically transfers the static inverter cabinet output to its associated GRID.
The normal input to the static switch is from the power switching circuit.
A second source of power is available from a non-1E 120 Vac constant voltage isolation transformer (alternate input).
If the static switch fault circuitry
detects a loss of normal input, load fault or overload condition, the faul't circuitry will initiate a bumpless transfer in < 1/4 cycle (4.17 msec maximum) to the alternate input.
(2)
Circuit Anal sis (a)
Inverter The inverter can be considered as having an input, output, oscillator, and power switching circuit (Silicon Controlled Rectifiers - SCRs).
The input is a filtered DC source that is capable of supplying transient-free power to the inverter SCRs.
The power switching circuit consists of four SCRs arranged in
'a full wave bridge configuration.
The oscillator generates gating pulses of the proper frequency to control the firing (turning on and off) of the SCRs to convert the DC input into a square wave output.
The resulting waveform is applied to the inverter output circuit which regulates (constant voltage transformer - CVT) and filters the output to provide a
sinusoidal inverter output.
Loss of inverter AC output, voltage (RL2B) is sensed at the output of the CYT and is annunciated (Annunciator Panel Number 219) in the main control room as
"CRID 4 Inverter Abnormal" and as a
failure light on the local static inverter control panel.
In addition, the following initiating devices also actuate the above annunciator:
Device Function PC6 PC7 PCS CB1, CB2 or CB4 RL21A SW1-C Static Switch Loss DC Input DC Input Overvoltage Loss of Alternate Source CRID Circuit Breaker(s)
Open Fan Failure Inverter Manual Bypass Switch In Bypass Transfer Switch Transferred The operator has the following local panel instrumentation available to assess static inverter operability:
Device Function V2 A2 Fl Vl Al DC Input Voltage DC Input Amperes Frequency Meter Inverter AC Output Voltage Static Inverter AC Output Amperes The Vl meter indicates the inverter output voltage between the inverter CVT and the inverter output breaker (CB2).
The Al meter indicates the static inverter output current
to the CRID and the meters associated current transformer also provides an input to the overload sense circuitry.
In addition, individual CRIO bus voltage indication is provided in the main control room.
(b)
Static Switch
/
The static switch is a high speed electronic switch that is used in uninterruptible power systems to automatically transfer a critical load from the output of a failed or overloaded inverter to an alternate source of power without interruption.
A fast acting reed relay is used to select which input source is to be applied to the load.
The normal input (inverter) is applied through two parallel, self biased, reverse connected SCRs (Nos.
209 and 210).
The alternate input is applied through the same type of SCR (Nos.
207 and 208) configuration.
Pressing the inverter-to-load pushbutton (PB201) wi 11 actuate the reed relay and apply gate pulses to SCR Nos.
209 and 210.
The SCRs will alternately conduct and provide a full wave sinusoidal output to the GRID.
Manual control is also provided such that pressing the alternate-to-load pushbutton (PB202) will initiate, via the reed relay, a static switch transfer to the alternate input.
(3)
GRID No.
4 Failure Anal sis/Loss of N-44 The loss of voltage to CRID No.
4 was the result of a high resistance cathode to anode failure of SCR No. 209.
As a
result, the CRID was exposed to a half wave sinusoidal voltage of approximately 84 Vac root mean square (rms) from the inverter-to-load output.
The vendor field engineer who was present for some of the troubleshooting and final restoration of the static inverter indicated this was not a typical failure mechanism of a static switch SCR, typically the SCRs fail shorted or failed to ground.
The static inverter did automatically transfer at the beginning of the event as described in Section 3 of this report.
The SCR failure caused the low CRID voltage which in turn caused NI Channel N-44 to blow its control power fuses (5A).
N-44 was probably a larger load and could not withstand the sharp increase in current prior to the static switch transfer.
At the same time, other fused equipment probably received heat damage to their fuses (without opening)
and were able to withstand the transfer to the alternate input as a result of the static inverter overload condition.
Investigation of the CRIO No.
4 transfer by an operator and I&C personnel determined the static inverter was normal as indicated by 120 Vac on meter Vl.
Permission was given to initiate a transfer back to the inverter input and PB201 was pressed.
The inverter-to-load light did not initially come on;
instead, the fan failure light came "on", the inverter failure light was dim and then went out, and finally the invertor-to-.load light. came "on".
As a result of the transfer, a reactor scram occurred, from the Loop 4 reactor coolant pump breaker position'reactor.trip,"-,numerous relays were chattering, indications and control functions powered from CRID No.
4 were lost, and an odor that was later attributed to overheated power supply transformers was detected.
The static switch should have initiated a second transfer to the alternate input if an overload condition'ad remained.
However, the static switch did not transfer as a result of CRID, No.
4 breakers opening and other equipment fuses opening.
This probably occurred as described above when the inverter failure light was dim and overload conditions were clearing.
The relay chattering remained until GRID No.
4 was transferred to unregulated control 'room power.
A failure of a static switch SCR is a nondetectable failure and the operator responded correctly when the transfer from alternate to normal input was made.
The inverter output voltage (120 Vac) indicated to the operator that the inverter was normal.
In addition, when the N-44 fuses were pulled, the operators believed that the bus overload condition had been caused by N-44 and therefore-had been cleared.
The opening of the N-44 control fuses was therefore an effect and not the cause for the loss of CRID No. 4.
The licensee repaired CRID No.
4 static switch and successfully tested the static inverter at. 90% (50A) of full load.
-Nominal static inverter output current is approximately 25A and several successful static switch input source transfers were performed prior to declaring the CRID operable.
Other Electrical Failures The following power supplies were damaged as the result of excessive transformer heating due to low GRID voltage conditions:
2-QDA-40 RCP No. 4, No. I Seal Differential Pressure Transmitter 2-NQ601 Miscellaneous Control and Indication Drawer 2-NQ602 The half wave AC input caused the power transformers to saturate, draw excessive current, and to overheat.
The outgassing of the transformer windi.ngs lacquer coating
.
produced the odor that was detected by the control room staff.
The licensee was in the process of replacing the three power supplies.
The comparator-rate drawer power supplies contained 5A control fuses.
Both control fuses were blown; however, the loading on
the fuses was probably low which resulted in damaging the power supplies prior to the fuses opening.
The 2-ADA-40 power supply is fed from a Balance Of Plant (BOP) bus that is supplied from GRID No. 4.
Isolation of the BOP bus from GRID No.
4 was provided by a Class 1E circuit breaker (20A).
The ADA-40 power supply was not provided with protective fusing. 'iscussions with the licensee indicated that this power supply was nonessential and its loss would have no impact on the plant.
The power supply was of a low current type (0.5A); even though protective fusing is a good practice, the transformer primary winding would probably have opened or was open and would have isolated the power supply from the bus without increasing the risk for a fire in the control room.
Loss of Control Room Instrumentation A total of six breakers tripped and a number of fuses were blown as a result of the CRID No.
4 low voltage condition.
See Attachment 4 for a complete list of instrumentation/control functions that were lost.
In all cases except steam generator (SG) wide range level, redundant instrumentation was available to the operator.
Regulatory Guide (R.G.) 1.97 recommends Category 1 wide range steam generator level instrumentation, with a range from the tube sheet to the separators, to monitor the operation of the steam generators.
The instrumentation provided by the licensee does not meet the environmental qualification criteria for Category I instrumentation as recommended in R.G. 1.97.
This instrumentation uses Class 1E power.
However, the same power source is used for all four wide range channels.
The licensee's position has been that Category I narrow range level instrumentation could be used to monitor heat sink capability.
If the water level is not within the range of this instrumentation, the licensee would use auxiliary feedwater flow to indicate the availability of the steam generators as a heat sink.
Although wide range steam generator level instrumentation is contrary to R.G. 1.97, the Office of Nuclear Reactor Regulation is reviewing this under the Safety Evaluation Report process.
The purpose of wide range steam generator level measurement is for the identification and mitigation of an accident and for determining the avai labi lity of the steam generator s as heat sinks.
This determination requires that the operator has a
positive indication of water level to ascertain the exact status of the steam generators at all times.
The loss of the single power source did cause the loss of all wide range steam generator level channels.
Environmental qualification has been clarified by the Environmental gualification Rule,
CFR 50.49.
The licensee.
should provide environmentally and seismically qualified
Category I wide range steam generator level instrumentation in accordance with the provisions of 10 CFR 50.49, R.G.
1.97, and R.G.
1. 100.
The licensee should also consider redundant Class 1E power, in accordance with R.G.
1.97, for this instrumentation.
Each SG is provided with one channel of wide 'range level indication and all four channels receive power from CRID No.
4.
Following the reactor trip, prior to SG level recovery by the auxiliary feedwater system, the narrow range SG level went off scale low.
The operators knew that wide range level would be lost if CRIO No.
4 failed and they used SG pressure, feedwater flow, and steam flow as an indication that the SGs were intact and recovering inventory.
Wide range level indication was available, at the Local Shutdown Instrumentation (LSI) panels located in the Auxiliary Building.
The LSI panels are provided with a transfer switch which could be used to select an alternate power source.
The licensee has performed surveillance tests (where applicable)
and functional tests on all the equipment affected by CRID No. 4.
The results were satisfactory and all the equipment/
components performed as designed.
(6)
CRID I II and III Review For Redundant Instruments The AIT randomly selected instrumentation and control loads that were supplied by the other three CRIDs.
The licensee reviewed the selected loads and provided the team with a list of redundant instruments and their CRIO power source.
The AIT reviewed the list and determined that the other CRIDs did not exhibit a design vulnerability similar to that existing in CRID No.
4 loads ( loss of all wide range SG level indication).
The licensee informed the team that they were independently reviewing all loads to the four CRIOs to ensure there were redundant instrumentation/control systems available if a GRID failed in the future.
b.
Control Rod Bottom Li ht Failure Following the reactor trip, the operators noted the absence of two rod bottom lights.
The licensee determined that two rod bottom bistables had failed to trip.
The root cause was a failure of a 100 microfarad capacitor in Shutdown Bank 'A'grid B4) bistable and a failed open resistor in Shutdown Bank grid C7) bistable.
The operators used other types of rod position indication and verified that the shutdown rods had inserted.
The bistables are calibrated during each refueling outage and AIT members reviewed the calibration procedure and determined that the procedure was adequate.
It is uncertain when the bistables could have failed; however, the failures appear to be random and not as a result of the CRID No.
4 failure.
The team verified that the power source to the rod position indication system was from Control Room Power-3 distribution bus which was unaffected by the CRID No.
4 failur c.
CRID Static Inverters and Control Rod Bottom Li ht Performance Histor (1)
The AIT reviewed the CRID computerized performance history on the static inver ters that has been compiled since 1985.
The review noted there were only two instances where corrective maintenance was performed on GRID No. 4.
One instance was to replace the oscillator board and the second instance was to adjust the frequency and voltage of CRID No. 4.
Minimal corrective maintenance has been performed on all of the CRIDs; in particular, none of the SCRs have been replaced.
The performance history indicates that the CRIDs have been fairly reliable and that the static inverter was able to transfer to its alternate source.
Thus, the licensee was adequately maintaining the CRID static inverters.
(2)
The AIT reviewed the rod bottom light bistable computerized performance hi story that has been compiled since 1985.
The review determined there was only one instance where a bistable module had to be replaced.
In this case, the rod bottom light was "on" at a rod position of 228 steps when it should have been "off". 'hus, the licensee was adequately maintaining the rod bottom light bistables.
d.
Turnin Gear Motor The turbine turning gear motor failed to operate properly during the coastdown of the turbine following the trip, and failed to operate properly following subsequent manual start attempts.
The motor experienced an electrical failure accompanied by heat and smoke.
There was no fire.
The motor failure proved independent of the reactor trip.
The turning gear motor is a 4160 volt, 1760 rpm motor, powered from 4KV bus 2B. It functions through reduction gears to turn the rotor shaft during startup and shutdown (and once per week while the turbine is idle) at 40 rpm, to prevent deformation of the rotor shaft.
This is a relatively severe service motor, and the licensee believes the fai lure was due to normal end-of-life wear.
After realizing the turning gear motor had failed, the licensee placed the turbine rotor on an air motor, which is used primarily during maintenance activities.
A spare turning gear motor was subsequently installed prior to unit star tup, and the damaged motor was sent offsite for repairs.
Main Feed Check Valve Initial information concerning the East Main Feed Pump inboard discharge check valve, valve 2-FW-103E was misunderstood by the NRC.
A postulated failure was considered potentially significant.
Should the valve malfunction in preventing backflow through the line, possible overpressurization of pump suction piping could resul l
The AIT learned that the licensee had questions and concerns about the valve just prior to the event, and was investigating possible further test or repair options.
However, the valve did not fail and properly performed its function both before and after the event.
The uncertainty regarding this valve concerned the inability to successfully complete a routine (weekly) test involving rotating the valve shaft slightly with an external linkage.
The shaft rotation normally "levers" on the disc, showing the disc is also free to move.
The most recent weekly test had determined that the valve shaft was not free to rotate.
The,"stuck" condition of the shaft was verified on August 13, 1989, and plant management was notified the morning of August 14.
The AIT reviewed this matter to determine the root cause of problems with the valve, including its maintenance and surveillance history, and evaluated the licensee's response to reports of the problem.
(1)
Determine the root cause of problems with the East feed pump inboard check valve 2-FW-103E:
.
The cause for binding of the valve shaft was determined to be a pinched/misaligned thrust bearing, probably from imprecise reassembly of the valve following a preventive maintenance (PM) activity performed during the recent Unit 2 steam generator replacement outage.
The PM included an internal inspection which revealed a slightly worn shaft thrust bearing and slightly scored packi'ng.
During restoration activities the new thrust bearing may not have been installed flush into its housing, which resulted in the bearing being "pinched" at one end.
The lack of procedure specificity on this installation was corrected.
(2)
Determine the maintenance and surveillance history of the check valve.
(a)
A review of, the maintenance hi story of this valve revealed that, aside from basic "clean and inspect" preventive maintenance and replacement of the shaft thrust bearing subsequent to the reactor trip, the only other work done on the valve was to repair the diaphragm in the spring loaded air cylinder external to the valve and repack the valve with a better packing material.
(b)
Surveillance test (OHI-5030 Test No. 95) records indicate testing of the check valve to insure proper operation was begun in March 1989.
The monthly (now weekly) test simply requires the operator to vent air out of the spring loaded air cylinder and observe the proper, positioning of the external rocker arm which indicates disk position.
Any abnormalities are required to be repaired by initiation of Job Orders.
The Maintenance job orders which were written
as a result of the surveillance test were initiated by operator s performing the test procedure.
(3)
Determine the adequacy of the licensee's actions on determining the valve was in a degraded state.
(a)
The design of this particular check valise is such that its closure ability was never affected.
The arrangement of the linkage connecting the check valve disc to the shaft is such that movement in the "open" direction is restricted only by a "stop" keyed to the shaft.
The purpose of the
"stop" is to insure positive closing of the valve by means of a spring assist device connected to the "stop" by suitable linkage.
(b)
When the check valve was disassembled on August 15, 1989, the check valve was found properly closed with a head of water against its disc.
(c)
Additionally, this check valve is the first of two series check valves downstream of the feed pump from which follow, in parallel, a motor-operated isolation valve and manually operated globe bypass valve, all of which functioned normally.
It thus appears that plant operation with this check valve in its degraded state had no detrimental effect on safe operation of the plant.
5.
Maintenance and Surveillance Practices The licensee was performing the following preventative maintenance (PM)
on the static inverters:
Frequency
~months Descri tion IS Clean Static Inverter Cabinets
Replace Capacitor C2
Replace Capacitor C805 120 Replace All Circuit Boards Replace Capacitor Cl Replace Fuses FU4 and FU5 Replace Fuse FU204 Replace I Pole Static Switch 192 Replace Fans The PMs appeared to be adequate
.for maintaining the continued reliability of the CRIDs.
The licensee does not perform any type testing to verify that the static switch will perform its intended function.
Actual static transfer events, such as this event, have occurred that demonstrate the static switch will perform as required.
However, the root cause of this event was an
undetectable SCR failure during the transfer from alternate input to the normal input.
The licensee presently has no mechanism for testing that.
the normal input SCRs-are operable
.
This failure mechanism could occur in any of the static switches which makes either of the D.
C.
Cook unit's CRIDs susceptible to this type of undetectable fai lure.
The licensee indicated to the AIT that they were looking into a method that would verify that the normal input SCRs were operable prior to transferring input sources.
Development of such a test should provide a means to prevent recurrence of this type of event.
6.
Procedures The AIT reviewed selected licensee procedures which were used leading up to the reactor trip or which were used in the reactor trip response and recovery.
Procedures relating to equipment maintenance and testing were also selectively reviewed.
A complete list of procedures reviewed, is included with this report as Attachment No. 5.
No significant deficiencies were identified in these procedure reviews.
Procedures did not appear to be materially involved in either causing the event or in complicating licensee response to the event.
Mhere procedure enhancements appeared prudent (e.g. specificity of reassembly instructions for valve 2-RIt-103E), they were accomplished.
7.
0 erator Res onses The AIT interviewed all of the operators and the IKC personnel who were present and involved in response to the reactor trip.
a.
~Back round As discussed previously, the loss of CRIO-IV placed the operators in an unusual situation that required recovering from a reactor trip with reduced indication and control. Although the operators had received simulator training on the loss of a CRID, the voltage degradation that CRID-IV actually experienced was unlike the instantaneous loss practiced in simulator drills.
Additional factors that had the potential to affect operator response were the many distractions that occurred simultaneous with the event.
At times, the chattering of bistables and relays was so loud that operators had to shout to communicate.
A burning smell led some operators to believe that there might be a fire somewhere in the control room.
The shift supervisor also had to contend with reported problems experienced on Unit 1, including a failed RTD indication and a smoking condensate booster pump.
Another distraction was a report of a Unit 2 turbine bearing fire (which turned out to be a smoking turning gear motor).
b.
0 erator Res onse The loss of CRID-IV occurred at 4:01 p.m., - just after shift turnover.
The operator who performed the CRID transfer, and the 18C
supervisor who was observing, realized shortly after the transfer attempt that something was wrong.
Normally, following a transfer from alternate to normal power the green indicating light (alternate supply) would go out and the red indicating light (normal supply)
would come on.
However, in this case, the green light went out and the red light failed to come on immediately after the transfer.
Instead, fan and inverter failure lights came in and went out before the red indicating light came on.
The operator and 18C supervisor at the CRID cabinet realized that the plant had tripped and immediately went back up to the control room.
In the control room the unit supervisor was reading through procedure E-O, "Reactor Trip or Safety Injection."
Step 1 of E-0 requires (1) verification that rod bottom lights are lit, (2) that reactor trip breakers (RTB's) are open, (3) that rod position indication is less than 25 steps and (4) that neutron flux is decreasing.
The reactor operator (RO) on the RCS/RPS panels noticed that two rod bottom lights remained unlit.
However, the reactor trip breakers were verified open, analog rod position indicated less than 25 steps on all rods, and neutron flux was decreasing.
The operators were trained that three of the four conditions of E-O, step 1 must be met, so operators considered the reactor tripped.
Because of the concern over the possibility that two of the rods were stuck between 0 and 25 steps, the RO requested permission to emergency borate.
Permission was granted by the unit supervisor.
Per E-O, Step 2, the balance of plant RO attempted to verify turbine generator trip.
However, the "stop valve closed" status lights were not lit, temporarily leading the RO to believe that the turbine may not be tripped. After manually initiating a solenoid turbine trip per E-0 and observing that the status lights were still not lit, the RO initiated the newly installed ATWS mitigation system actuation circuitry (AMSAC).
There are no procedures in place at D.C.Cook for initiating AMSAC, and to date the operators had not received simulator training, yet the RO had remembered from his classroom training that AMSAC tripped the turbine.
The RO did not.realize that the AMSAC turbine trip utilized the same mechanism as the manual solenoid turbine trip.
Turbine trip was verified by other means such as stop valve limit switch position.
Initially, the loss of CRID-IV required the balance of plant operator to verify and maintain adequate heat sink without the benefit of steam generator level indication or steam dumps.
Steam generator level indication was unavailable because the narrow range level indication was offscale low (for about 15 minutes)
as the result of post trip shrink, and all four SG wide range level indications are supplied by CRID-IV.
The RO verified adequate heat sink by observing auxiliary feedwater flow indication (except for auxiliary feedwater flow to SG No. 4, which was powered from CRID-IV) and SG pressures.
Although they admitted that they would have preferred to have SG levels, none of the operators interviewed indicated a concern about not having steam generator level indication in response to this event.
Pressure control (other than code safeties)
was automatically available only with the No.
2 and No.
SG power operated relief valve (PORV), since the No.
1 and No.
SG PORV's receive control power from CRID-IV and a steam dump interlock fails low on loss of CRID-IV.. Manual control was available on all 4 PORYs.
The RO operated the SG PORV's in manual to aid in plant cooldown.
The No.
PORV was left closed after RCP No.
4 had been secu'red.
According to the assistant shift supervisor, the high auxiliary feedwater flow rate significantly reduced the need for the SG PORV's.
In fact, the operator s were concerned about the possibility of an inadvertent Safety, Injection (SI) signal from the cooldown because, with the loss of power from GRID IV to the Train B Solid State Protection System (SSPS)
output relays, response to an SI signal (verification, reset, termination) could be difficult.
Several actions taken, which were not explicitly required by procedures, were said to be very helpful to the operators.
For example, one of the operators read through an attachment to an operating procedure (OHB 4022.082.001)
which identified the valves, instrumentation and control functions provided by CRID IV.
There was also evidence of prudent actions on the par t of the operating crew to obtain independent confirmation of plant status parameters.
For example, the Unit Supervisor requested an Auxiliary Equipment Operator (AEO) to verify Component Cooling Water (CCW) flow to the Letdown Heat Exchangers from indication outside of the control room.
About 40-50 minutes after the trip GRID-IV was transferred to lighting panel CRP-3 and the steam dumps and other indication and control from CRID-IV were subsequently restored.
c.
Conclusion Despite the considerable loss of control and indication in the control room as the result of losing CRID-IV, the operators demonstrated that they could safely recover from the reactor trip.
Operators appeared to properly follow procedures and take appropriate actions.
Several instances were noted where operators went beyond procedures in taking appropriate and conservative corrective actions.
Licensee Mana ement Res onse and Re ortin At the time of the event, the Plant Manager was not onsite; he was in the license's Corporate office in Columbus, Ohio.
The senior manager onsite was the Assistant Plant Manager (APM)-Production, whose day-to-day responsibilities include, the operations and maintenance functions.
The APM-Administration was on vacation but physically present at his home near the site.
The APM-Production was notified of the reactor trip shortly after it occurred and he went to the Unit 2 control room to monitor plant recovery.
He notified the NRC Senior Resident Inspector (SRI) onsite within about 15-20 minutes of the event, and briefed the SRI upon his arrival at the control room a few minutes later.
Formal notification to NRC via the Emergency Notification System (ENS)
was completed at 7:41 p.m., within the 4-hour time limit of 10CFR50.72.
This notification was a brief summary of the event including actions leading up to the reactor trip and a few of the indications and controls lost with CRID-IV.
At the time of this notification, the licensee investigation of the loss of the CRID and associated instrumentation was continuing.
Licensee management onsite, in consultation with managers in Columbus, Ohio, made an early decision that plant restart would require review by the onsite Plant Nuclear Safety Review Committee.
Thereafter, a planning meeting was conducted at about 9:00 p.m.
on August 14 to review information collected on plant conditions and status.
This meeting determined'that completion of necessary repairs and evaluations for restart was not assured prior to elapse of a 30-hour Limiting Condition for Operation (LCO) in,effect as a consequence of the CRID failure.
Therefore, instructions were issued to cool down the reactor coolant system, beginning after shift change about midnight August 14.
This was a
conservative decision.
Onsite senior managerial responsibilities were formally turned over to the APM-Administration effective late on August 14..
This manager had reported to the site as backup to the APM-Production, who had a prior commitment to travel to corporate headquarters on August 15.
The APM-Administration was present for the August 14 planning meeting, and remained in charge through August 15.
On August 15, plant status (8:00 a.m.)
and schedule review meetings (1:00 p.m.) were conducted at the site with managers at the corporate office connected via teleconference.
The NRC SRI attended both meetings and briefed NRC Region III.
Licensee decisionmaking appeared prudent considering the information available.
Also, at about 11:00 a.m.
on August 15, a teleconference was conducted among site and corporate licensee personnel and'site and Region III NRC personnel.
The licensee's management was forthcoming and cooperative in agreeing to provide NRC Region III with a letter documenting actions taken or planned in response to the event.
By this time, general direction had also been given to the plant, from the Plant Manager, to
'pproach equipment troubleshooting and repair carefully and to preserve failed components for later evaluation.
The identification and sequestering of these components later proved useful to the NRC AIT.
The licensee contacted the CRID-IV vendor, Solid State Control, Inc.,
on August 15, and made arrangements for a technical expert to come to the plant site to support the investigation and repair processes.
The vendor representative arrived onsite on August 16, and also proved useful to the NRC AIT.
Overall, the NRC AIT found the licensee's management response to be conservative and cooperative, and communication of information to NRC met requirements.
Items Re uirin Additional Res onse or Ins ection Based on the AIT review, the licensee was,tasked with the following items, the results of which will be made available to the NRC upon completion.
'
(a)
Confirm the failure mode of SCR 209.
In this regard, the licensee has retained their contractor (SCI) to confirm the failure mode of the SCR and to determine if the bent pin found on SCR 209 was or was not a contributor to failure.
(b)
Review the loads supplied by CRIDs I, II and III to determine if there are other loads which, upon failure of the CRID, would cause a loss of all channels of a indication to the control room operators (such as steam generator wide range level indication with failure of CRID IV).
(c)
Determine the feasibility of a preventive maintenance/surveillance procedure to check the operation of the static switch and its power supplies prior to transfer from the alternate to normal power supply subsequent to an automatic transfer from normal to alternate.
If such a test is feasible, the licensee will also consider incorporating this test into a periodic test possibly performed every refueling outage.
(d)
In addition to the above, the following items will be reviewed in a subsequent inspection to determine if the licensee's corrective actions to these items are acceptable to Region III and meet appropriate regulatory requi rements:
(1)
Guidance and/or training provided by the licensee to plant operators regarding the use of the ANSAC system.
(2)
The maintenance program and the training to maintenance mechanics with regard to the maintenance activities performed on check valve FW 103.
10.
Exit Interview 30703 The inspectors met with licensee representatives (denoted in Paragraph 2)
throughout the inspection and evaluation of the events and at the conclusion on August 17, 1989, and summarized the scope and findings of the augmented inspection team's activities.
The licensee acknowledged these findings.
The inspector also" discussed the likely informational contents of the inspection report with regard to documents or processes reviewed by the inspector dur ing the inspection.
The licensee did not identify any such documents or processes as proprietary.