IR 05000461/1989021
| ML20247N316 | |
| Person / Time | |
|---|---|
| Site: | Clinton  |
| Issue date: | 07/14/1989 |
| From: | Ring M NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III) |
| To: | |
| Shared Package | |
| ML20247N282 | List: |
| References | |
| 50-461-89-21, CAL-RIII-89-16, NUDOCS 8908020357 | |
| Download: ML20247N316 (20) | |
Text
__
.
,
,
c
-
.
.
i U. S. NUCLEAR REGULATORY COMMISSION
REGION III
.
Report No. 50-461/89021(DRP)
E Docket No. 50-461 License No.'NPF-62 Licensee:
Illinois Power Company 500 South 27th Street Daratur IL 62525 Facility Name:.Clinton Power Station Inspection At: Clinton Site. Clinton, Illinois
-
Inspection Conducted: June 1,1989, through June 23, 1989 Inspectors:
P. Hiland S. Ray J. Jacobson H. Walker P. Brochman W. Luckas 7 't'/97 Approved By: Mark A. Ring, Chief a
Reactor Projects Secti n 3B Date
,
Inspection Summary Inspection on June 1, 1989, through June 23, 1989 (Report No. 50 401/89021(DRP))
Areas Inspected: Special safety inspection in response to i Tallure of the
"B" reactor recirculation pump seals and subsequent equioment problems which occurred on June 1, 1989.
Results: Of the six areas inspected, one violation was identified concerning a procedural violation which resulted in a recirculation pump seal pressure instrument being isolated during plant startup and system pressurization (Paragraph 5.f.)
In addition, several weaknesses were identified in the area of Emergency Response Organization performance (Paragraph 4.)
8908020307 890724 ADOCK0500g.gl PDR O
. _
_ _.
_ - _
_.
- _ _ _ _ _ _ _ _ _ _ _
p. '
Jf:.
!
'
.,
.
,
.
..
'
DETAILS I
1.
' Personnel Contacted
' Illinois Power Company (IP)
h D. Hall, Senior Vice President
'
- J. Perry, Assistant Vice President.
- J. Wilson, Manager - Clinton Power Station R. Freeman, Manager - Nuclear Station Engineering R. Campbell, Manager - Quality Assurance D. Holtzcher, Acting Manager - Licensing & Safety huclear Regulatory Commission P..Hiland, Senior Resident Inspector, Clinton
- S. Ray,' Resident Inspector,.Clinton H. Walker, Reactor Inspector, Division of Reactor Safety, RIII J. Jacobson, Reactor Inspector, Division of Reactor Safety, RIII P. Brochman, Senior Resident-Inspector, Byron W.- Luckas, NRC Consultant,. Brookhaven National Laboratory
- Denotes those persons attending the exit meeting on June 23, 1989.
The inspectors also contacted and interviewed other licensee and
. contractor personnel.
2.
. Event Synopsis On. June'.1, 1989, at 12:44 a.m., with the reactor at about 42% power during power ascension following a refueling outage, the "B" reactor recirculating'(RR) pump seals failed. At 12.:55 a.m. an UNUSUAL EVENT was declared when drywell floor drain leakage exceeded five gallons per
. minute. At.1:00 a.m. an ALERT was declared when drywell floor drain leakage exceeded 50' gallons per minute. Foilowing isolation of RR Loop-B and isolation of rod drive seal water supply to'RR Pump-B, the drywell floor drain leakage returned to a normal level and the event was downgraded to an UNUSUAL EVENT at 1:40 a.m.
Early in the event, drywell chiller-A tripped on low condenser water flow. Drywell pressure increased to a maximum of 1.36 psig and the combustible gas control system (CGCS) compressors were started at 1:05 a.m. to reduce drywell pressure. At 1:10 a.m. drywell chiller-B was started and the CGCS compressors were secured.
At 1:40 a.m., with the plant operating in " single loop" a normal reactor shutdown was initiated. At 3:25 a.m. the UNUSUAL EVENT was terminated and the plant sSutdown continued. At 6:10 a.m. a manual scram was inserted from 3% reactor power when the motor driven feedwater pump regulating valve.(FRV) failed to respond to control signals.
i
___
- _. - _ - _ _ _ - _ _ _
a
.y i
i
.
.
.
..
'
Based on'the above events,' actions to be taken by the licensee were detailed in NRC. Confirmatory Action Letter, CAL-RIII-89-016, dated June 1, 1989, and-included the following:
"1.
Perform an evaluation as to the cause of the failure of both seals on the RR pump, the inability of the FRV [ Feed Regulating Valve] to control ~ level and the trip of the drywell chiller. Any systems or components which did not function properly will-be included.
..
' 2. -Identify _any equipment that malfunctioned and place such equipment into a. quarantine status until the NRC Special Inspection Team has
- the. opportunity to review your action. plan regarding this equipment.
This would include any equipment so designated by the Team Leader (P.Hiland). The RR pump seals and the FRV would be expected to be included in the quarantine. Removal of the seal package from the RR pump is authorized provided that no disassembly of the seal package is performed.
3.
Evaluate the equipment history of those components believed to have malfunctioned and include in the quarantine any previously replaced components as are still available.
4.
Except as dictated by plant safety, advise the NRC special team
- leader prior: to conducting any-troubleshooting activities.
Such notification should be provided soon enough to allow time for the
. team _ leader to assign an inspector to observe activities.
5.
Submit to NRC Region III.a formal report of your findings and conclusions'within 30 days of receipt of this letter."
-
Further, the licensee was to obtain concurrence from the Region III Regional Administrator prior to restart of the Clinton Power Station.
On June 16, 1989, the licensee met with Regional Management to discuss
~
the status of the actions discussed in the CAL. Regional NRC Management
-
outlined'the actions they understood the licensee would take during the plant startup and the Regional Administrator gave his concurrence for the restart. The NRC issued a_ letter to the licensee on June 21, 1989, confirming those discussions and understandings.
~ 3.
Plant Staff Response a.
General The inspectors' review of the licensee's response to the e/ents on
,
,
June 1,'1989, consisted of direct observations of the operating crew's activities from about 3:00 a.m. to 7:00 a.m. on June 1,1989, attendance at a critique of the event on June 2,1989, interviews with management and plant staff personnel, review of applicable
logs, chart recordings, computer alarm printouts, General Electric Transient Analysis Recording System (GETARS) traces, and procedures,
>
l
= - _ _ _ - _ _ _ __._
_ - _ _ - - _ _ _
-
.
.
.
-
.
'
b.
Sequence of Events At the time the 11:00 p.m. to 7:00 a.m. shift operating crew assumed control of the plant on the night of May 31/ June 1, 1989, the plant was operating at about 36% power with both RR pumps operating in slow speed. One of the objectives of the crew was to increase power to 50% by about 1:00 a.m. so there would be time to perform surveillance that were required within 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> of increasing power above 25%.
In order to increase power to 50% it was necessary to
' shift RR pumps to fast speed in accordance with Operating Procedures CPS No. 3004.01, " Turbine Startup and Generator Synchronization,"
Step 8.4.2, and CPS No. 3302.01, " Reactor Recirculation," Step 8.1.2.1.
The crew was aware of a problem that had been observed with the
"B" RR pump seals on May 25, 1989, and discussed the possibility of seal failure upon shifting the pumps to fast speed. They discussed the indications of seal failure and reviewed the actions they would take if failure was indicated.
'
At 12:41 a.m. the "A" RR pump was shifted to fast speed with no abnormal indications. At 12:44 a.m. the "B" RR pump was shifted to fast speed. The operators noted indications of seal problems
'immediately after the shift to fast speed. Upper seal cavity staging pressure dropped from 475 psig to 38 psig. At 12:47 a lower seal cavity high temperature alarm was received along with seal staging flow, outer seal leakage, and seal cooling water temperature alarms. Lower cavity pressure decreased from 952 psig to about 900 psig and was observed to be oscillating and dropping further.
Drywell floor drain sump inlet flow was observed to be increasing from about three gallons per minute.
At 12:50 a.m. the drywell floor drain sump flow alarmed at five gallons per minute. At 12:53 a.m. both RR pumps were shifted to slow speed. At 12:55 a.m. the Shift Supervisor declared a NOTIFICATION OF UNUSUAL EVENT (NOUE) in accordance with Emergency Plan Implementing Procedure EC-02, " Emergency Classification," and the "B" RR pump was secured. At 12:59 a.m. the "B" RR loop was isolated. Leakage flow from control rod drive water to the RR seals continued for a few minutes until the manual supply valve in the containment was closed.
At 1:00 a.m. drywell floor drain sump inlet flow reached 50 gallons per minute and the Shift Supervisor upgraded the event to an ALERT in accordance with EC-02. At about 1:10 a.m. the drywell floor drain sump flow began to decrease because the seal leak had been isolated. At 1:40 a.m. plant conditions were stable with drywell sump flow at about seven gallons per minute and the Shift Supervisor i
downgraded the emergency classification to NOUE.
The crew started a plant shutdown in accordance with Operating Procedure CPS
'
No. 3006.01, " Unit Shutdown," on orders of the Manager - Clinton l
Power Station. At 3:25 a.m., after verifying that no radiological release had occurred and that conditions in the containment had
returned to near normal the Shift Supervisor secured from the NOUE.
'
l
l L___.__.--
_.
-
_
.
i
,
i
,
.
.
I
.
Changes in parameters of interest during the event included the i
following:
~
Drywell pressure increased from near 0 psig to 1.36 psig.
r
- Drywell fission product gaseous activity increased from 50 cpm to 200 cpm.
Drywell fission product particulate activity increased from 100 cpm to 3000 cpm.
Drywell floor drain sump inlet flow increased from 1 gpm to slightly over 64 gpm (off scale).
"B" RR seal cooling water discharge temperature increased from 91-degrees Fahrenheit to greater than 300 degrees Fahrenheit (off scale).
"B" RR lower seal cavity temperature increased from 124 degrees Fahrenheit to 208 degrees Fahrenheit.
"B" RR upper seal cavity temperature increased from 116 degrees Fahrenheit to 193 degrees Fahrenheit.
"B" RR motor lower guide temperature increased from 116 degrees Fahrenheit to 153 degrees Fahrenheit.
Early in the ;eal failure event the operators noted that drywell pressure was increasing. They noted that the "A" drywell chiller unit which had been running had tripped. Operators were dispatched to the chillers to determine the cause of the trip and start the standty "B" drywell chiller. At 12:56 a.m. operators reported that the."A" chiller had tripped on condenser low water flow (service waterinletpressure). Drywell pressure was increasing toward the Emergency Core Cooling System (ECCS) actuation setpoint of 1.68 psig.
/J, 1:00 a.m. the operators started both hydrogen mixing compressors with drywell pressure at about 1.36 psig. The compressors removed air from the drywell and discharged it to the suppression pool. Drywell pressure started to decrease. At 1:10 a.m. the "B" drywell chiller had been started and the hydrogen mixing compressors were secured.
Because of the possibility of the compressors discharging activity to the containment, the Shift Supervisor ordered the containment evacuated and directed Radiological Protection personnel to perform surveys. The Containment Evacuation Alarm was sounded.
l c.
Evaluation of Operator's Actions The inspectors determined that the actions of the operating crew were generally good. Applicable procedures were referred to, personnel resources were used effectively, timely reports were made, and logs were accurately kept.
_ _ _ _ _ - _ -
--
- __ -. __
_ __
- _.
_
._-
'v'.,
-
.
.
,;
.
'
~
The "B" RR' pump ran for a total of about nine minutes in fast speed and two additional minutes in slow speed before being tripped. The operators were aware immediately after shifting the pump to fast speed that there was a problem with the upper seal. Operating Procedure CPS No. 3302.01, " Reactor Recirculation," did not require-
. immediate tripping of the pump upon high seal. temperature or flow, Lor low seal staging pressure. Management. direction during a
.
previous degradation of'the "B" RR seal in December 1988, had been
"
to trip the pump if unidentified leakage (as indicated.by drywell floor drain sump inlet flow) reached five gallons per minute.. In this case, earlier tripping of the pump may have reduced the mechanical damage, and total quantity of water which leaked, but the operators actions were in accordance with existing procedures and management direction.
The operators shifted the RR pumps to slow speed after they had
indications of drywell floor drain sump flow greater than five gallons per minute and increasing. The "B" RR pump was tripped shortly after indications of lower seal failure when the lower-seal staging pressure rapidly dropped below 900 psig after a few minutes of oscillating around 900 psig.
Due to the increasing drywell pressure, the operators were reviewing Off-Normal Procedure CPS No. 4402.01, " Containment Control-Emergency." Step 3.2.2 of that procedure instructed the operators to start one or two CGCS compressors if drywell pressure exceeded 1.68 psig. At 1:00 a.m., with drywell' pressure at 1.36 psig and increasing, the operators decided to start both CGCS compressors.
The operators stated that procedures did not recommend starting the
'
compressors until after drywell pressure reached 1.68 psig because starting them sooner might mask a small break Loss of Coolant Accident (LOCA) and prevent a necessary actuation of the ECCS at 1.68 psig.
In this case, however, they knew the cause of the L
pressure increase (RR pump seal failure and loss of normal drywell cooling) and had the situation under control (RR loop being isolated and standby.drywell chiller being started.) They stated that allowing drywell pressure to increase to the point of ECCS l-initiation would have unnecessarily complicated the event. The l
inspectors noted that there was no procedural guidance for l-controlling drywell pressure increases until after an ECCS initiation but that the operators' decision to use the CGCS compressors was reasonable. The CGCS compressors were secured as soon as the standby drywell chiller was started. As discussed in Paragraph 3.d below, the crew was aware that running the CGCS compressors had the potential to spread contamination from the
,
drywell to the containment, but took adequate precautions to prevent personnel contamination and assess the radiological conditions.
'
Personnel on shift were used effectively and priorities were established.
In addition to the normal personnel, two trainees were on the shift at the time. While the normal personnel were used for equipment operation, the trainees were asked to leave the horseshoe
_ _ _ - - _ _ _ _
. - -... - - _.
I
p
.
-
.
,
.
p
'
area and were use'd to monitor drywell and RR pump chart recorders as well as to monitor emergency classification procedures for entry conditions into the various action levels, d..
Radiological Response The response of radiological protection (RP) personnel to the event war, good. After starting the CGCS Compressors, the main control room operators sounded the containment evacuation alarm.
The RP Shitt Supervisor (RPSS) directed security to restrict access to
'
containment and allow only individuals who had the concurrence of the RPSS or Operating Shift Supervisor to enter. The standby gas treatment system (SGTS) was running for a routine surveillance so RP personnel closely monitored the SGTS stack radiation monitors for indications of containment leakage and offsite releases.
Until the radiological situation in the plant was determined, the'RPSS ordcred i
that no work be performed in the power block without his concurrence.
RP Technicians were ordered to monitor all available area and process radiation detection instrumentation.
As a result of the ALERT emergency classification, the Emergency Response Organization was called in to man the Technical Support Center (TSC). Ten RP personnel responded to the TSC with an additional ten personnel standing by to form five off-site monitoring teams. Although not used, the teams had all the necessary equipment to perform off-site dose assessments.
At 3:25 a.m. on June 1,1989, based on verification that radiological conditions in the drywell had returned to near normal and that all monitors and samples indicated no off-site or in-plant releases, the UNUSUAL EVENT was secured. Monitoring of the
'
containment continued until normal access was restored at 5:09 a.m.
Because the CGCS discharged air from the drywell into the containment suppression pool, most of the activity was scrubbed by the water and no measurable increases in dose rate or contamination levels in the containment were noted.
e.
Plant Shutdown At 1.:40 a.m. on June 1, 1989, the operators started a normal plant shutdown with the "B" RR loop isolated using Operating Procedure CPS No. 3006.01, " Unit Shutdown." The shutdown proceeded normally through securing the "B" turbine driven reactor feed pump (TDRFP),
securing the main turbine and generator, and entering OPERATIONAL CONDITION 2 (Startup) at 4:46 a.m.
Step 8.1.4.a) of CPS No. 3006.01 noted that if cool down was not desired or decay heat load would not support the steam loads, feed should be shifted from the TDRFP to the motor driven reactor feed pump (MDRFP) to minimize cool down. The operators were aware of a previous history of problems with controlling feed using the MDRFP feed regulating valve (FRV) but elected to attempt to use the MDRFP since decay heat was low. The plant had recently started up from en
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-
F" f
'
.
,
,l',
'
r
.
--
..
.,
I
'
extended refueling ostage and had little power history. ~ The
,
operators 1tated that they did not want to drive. rods to bring the
,-
reactor subcritical at the same time that;a TDRFP was using steam.
The. inspectors concluded that, under the circumstances,-the decision
.to use the MDRFP was reasonable.
At 5:54 a.m. the operators started the MDRFP and started backing the H
flow 'down on the running "A" TDRFP. :The FRV apparently did not-l initially respond-and reactor _ level dropped approximately 10. inches over a 2-3 minute period. The FRV then responded and appeared for
'
E
- 2-3 minutes to be operating properly. : Reactor level recovered but -
then continued to rise. Attempts to control ~the level increase with the FRV failed. After about a 20 inch increase, the operators closed both reactor inlet valves (IB21-F065A and B) to isolate all feed to the reactor. When reactor level. returned to normal, the
. operators attempted to control level by throttling one of the
!-
reactor inlet valves..This technique was not proceduralized but had
_been used successfully.in the past. Operating Procedure CPS-
,
No. 3103.01, "Feedwater," Section 8.2.4.2 gave directions for
-
- operating with a loss of control signal to the FRV. That procedure directed the operator to take manual control of the MDRFP minimum
. flow valve (FW010C) to divert feed flow to the condenser while starting a TDRFP. That technique was not used. Discussions with the operators indicated.that, in their experience, use of FWO10C would not have been adequate to stop the level increase.
' After about 1 minute of attempting unsuccessfully to control reactor level-by throttling a reactor inlet valve, the reactor operator manually scrammed the reactor from about 3% power' as level
, approached the Level 8 trip setpoint. 'The-plant's response to the
' scram was normal ~ and the operators properly carried out Off-Normal Procedure CPS No. 4106.01, " Reactor Scram."
The inspector was.in the main control room from the time the MDRFP was started until the plant was stabilized after the scram.
Later
. review of GETARS traces for the event indicated that the FRV was receiving only intermittent control signals and freezing "as-is" when the signals were not getting through. This problem is discussed more-fully in Paragraph 6 of this. report.
.4.
Emergency Response Organization Performance
- The~ inspectors reviewed the licensee's actions to verify that the Clinton Power Station Emergency Plan Implementing Procedure (EPIP) had been properly followed during this event. The inspectors reviewed log records,; notification forms, and emergency plan implementing procedures.
t xThe inspectors reviewed EPIP EC-02, " Emergency Classification," and verified that the licensee had properly classified this event as-ALERT when the primary coolant leakage rate was greater than 50 gpm. The licensee notified the State of Illinois and the NRC within the required-times. However, a. review of the logs and interviews of control room e
_
--_..--
_ _ _ _ _ - _ _ _ _ _ - _ _ _ _ _ _ _ _ _ - _
__ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ - _ _ _ _ - _ _ _ _ _ _ -
_
-
.
.
.
,
'
.
operators indicated that the plant general purpose alarm was not sounded
- at 1:00 a.m., in accordance with EPIP EC-04, Paragraph 4.1.1.
t.
,
o
- Additionally, a written summary of events was not provided to the NRC within 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />, in accordance with EPIP AP-03, Paragraph 4.3.1, but was
<
submitted 6 days later.-
.
Several problems occurred when the licensee implemented EPIP EC-07,
" Emergency Plan Notification." When an ALERT was declared, pager groups
- A, B, and D were to be activated. Only pager group A was activated by security personnel. Second, the autodialer failed when personnel attempted to activate it, possibly due to the voltage transient on the 6.9 KV busses after the recirculation pump shifts. Since a voltage
'
transient could be routinely expected during most major events, the licensee needed to review the susceptibility of the autodialer to voltage transients.. Persor.nel who were paged were required to contact the load dispatcher, who then kept a record of who had responded to the page.
This record was used to determine that all required disciplines would have an adequate number of personnel or that additional augmentation'was required.
In this event, there was confusion regarding whether the dispatch center was keeping the record.
Control room operators reported that they had been contacted by several members of the emergency response team to ask if the ALERT was real and-
'
to request additional information. This was a significant weakness from both the aspect of team members not believing the notification system and o
interfering with control room operators as they were trying to
- investigate the event.
In reviewing the official logs of this event, the inspector had difficulty in determining certain facts without relying on personnel interviews. The inspector recommended that the licensee improve its log
' keeping ability and training on the use of logs.
It was unclear to the inspector whether the problems which occurred with notification of the emergency response team members were promptly communicated to the Station Emergency Director, so that any necessary adc'itional actions could be taken.
From an overall perspective, the licensee's response was acceptable; however, several concerns were identified which required action by the licensee to resolve. The inspectors reviewed the licensee's critique of the Emergency Response (LS-89-0054) dated June 12, 1989. The critique described 19 problem areas identified in the response along with the causes and completed or scheduled corrective actions. Clinton Commitment Tracking forms were issued for all uncompleted corrective actions. All I-of the inspectors' concerns described above were addressed in the critique and resolutions have been included in the corrective actions.
Corrective actions from this event will be reviewed in a subsequent inspection by specialists from the Region III office.
'
l:
l l
- _-
,
.
.
.
' 5.
Failure of Reactor Recirculation Pump Seals The reactor recirculation (RR) systems at the Clinton Power Station were originally designed and installed utilizing type "A" RR pump seals in both the A and B loops. Subsequent to the installation, a new RR pump seal (type "C") was developed. This seal was considered to be far superior to the seals installed at Clinton and new type replacement seals were ordered and received. Maintenance Work Requests (MWRs) for installation of the new type seals were issued on September 23, 1985.
Since the installed type
"A" seals appeared to be functioning properly and a plant outage was required for RR pump seal installation, the MWR was considered to be low priority and was categorized as a job priority 5.
Starting in late November, 1988, a gradual degradation of the seals on the "B" RR pump was noted.
Plant operators and engineers monitored the seal parameters closely and discussions were held with vendor representatives. On December 17, 1988, seal leakage approached five gallons per minute indicating acceleration of the degradation and the "B" RR pump was stopped. The "B" loop was isolated and the plant operated at reduced power until the first refueling outage which began on January 2, 1989. The "B" seal was replaced on January 23, 1989, early in the outage. Disassembly of the "B" seal indicated normal deterioration and failure of the seal. Post maintenance testing was performed as part of the pressure testing of the reactor vessel following refueling activities on March 19, 1989. The
"B" seal was determined to be leaking and again was replaced. A second pressure test was successfully conducted to determine that the seal did not leak. Due to higher priority work during the outage, the removed seal was not disassembled promptly and the cause of failure was not immediately determined.
The pump was run almost continuously from April 10, 1989, to June 1, 1989, on slow speed. On May 25, 1989, pressure decreased in the seal outer cavity to approximately 60 psig indicating failure of the second seal stage. Approximately ten hours later the seal appeared to reseal and operated normally. On June 1, 1989, when the RR pump was switched to fast speed, gross leakage (greater than 50 gallons per minute) through the seal occurred and plant shutdown commenced.
The disassembly of the previous RR pump seal which failed on March 19, 1989, was performed on May 30, 1989. The cause of failure was determined to be improper installation of the stationary seal rings. The rings were found to be installed upside down. The licensee stated that the extended delay in the disassembly and cause of failure determination was due to higher priority work during the outage and the opinion by some individuals that the leakage was due to an "0" ring failure.
An action plan was developed for repair and failure analysis of the June 1, 1989, RR pump seal failure. The inspector reviewed the action plan and implementation.
The action plan required the removal of the failed seal and the assembly and installation of the spare seal assembly. The plan also required the disassembly and evaluation of the failed seal assembly to determine the cause of failure.
Specific actions required by the plan were as follows:
- - _ - _ _ - _ _ -.
_ _ __ _ --
- _- - - _ _ _ _ _ -
- _ - - - - - -
-_
--
--_ -_
!
D L~
.,
,
,
i
'
a.
Procedure Review - A review of Maintenance Procedure CPS
<
No. 8225.01, " Reactor Recirculation Pump Seal Removal, Installation i
and Maintenance," was required to ensure appropriate vendor manual instructions had been' incorporated.
j
!
The licensee completed that review and a number of minor changes were made' including a clarification to ensure that seal rings were installed properly. Revision 6 was issued on June 2, 1989. The inspector reviewed Revision 6 after issue and noted the changes.
Revision 6 contained additional instructions for the installation of the stationary seal rings. These' instructions should preclude the improper installation of the stationary seal rings which resulted in
!
the failure of "B" RR pump seal on March 19, 1989. The inspector
- noted there was no sign off required for the completion of this critical step. This concern was discussed with the licensee.. The i
.
inspector was told the concern would be noted for inclusion in the procedure at-the next revision.
b.
Parts - Parts to be used in seal rebuilding were to be verified by Quality Verification prior to use.
!
Quality Verification of.the parts to be used in the rebuild of the i
spare seal was completed on June 2,1989. The inspector witnessed a portion of this verification.
c.
Disassembly - After removal, disassembly of the "B" RR pump seal was to be' completed and evaluated to determine the cause of failure.
The inspector witnessed the removal of "B" pump seal by closed circuit television. Closed circuit television was used due to limited space and radiation exposure considerations. Difficulty was j
encountered in removing the coupling spacer, however, the seal
!
package was removed with little difficulty.
After removal, the inspector witnessed the disassembly of the seal package.
Extreme difficulty was encountered in the activity.
Disassembly was completed on June 6, 1989. The sealing surface of i
the rotating seal ring of the lower seal was broken in eight to j
ten pieces. The stationary seal ring of this seal contained surface d
cracks and scores but appeared to be in one piece. Both the rotating and stationary seal rings of the upper seal were completely destroyed. Only a few small pieces of the seal rings remained.
Other parts of the seal assembly were severely damaged and could not
)
be reused.
d.
Quality Verification of Replacement Activities - Quality Verification was to verify the assembly and installation of the replacement seals.
The inspector witnessed the assembly of the replacement seal.
Quality Verification was present and participated in these activities.
_
_
__
- - - - - - - -
_ _ - - - - - - - - _ _ - - -
- - - - - - - - _ _ - - - _ _ _. - _ _ _ _ - - _ _ _ _ - - _ - _ _
_________J
.
_- - -
,
I s
x
-
. -
, -
,
The assembly was performed per MWR #D02472. This MWR appeared to f contain adequate instructions for the assembly. Actual assembly was performed per Revision 6 of CPS procedure No. 8225.01. A consultant was present during the assembly. Craftsmen and the consultant appeared to be knowledgeable of the work to be done. Procedure compliance. appeared to be adequate.
e..
'!A" RR Pump Seal - A decision was made to remove and overhaul
"A" RR pump seal.
Since the evaluation for cause of failure of the "B" RR pump seal was inconclusive, the licensee decided to remove and disassemble the-
"A" pump seal.
"A".RR loop, including the pump seal, had operated normally since restart from the refueling outage.
Removal and disassembly of the "A" RR pump seal was completed on June 9, 1989.. The inspector witnessed the disassembly and noted that it came apart easily with no binding. All parts appeared to be nonnal and no problems were noted that would help in determining the cause of'"B" RR pump seal. failure.
The replacement seal, which was originally assembled for installation in
"B" RR pump, was installed in "A" pump and pressure tests indicated it was functioning normally. The seal package removed from "A" RR pump was reworked and-installed as a replacement for.the failed "B" RR pump seal.
f.
Cause of Failure Evaluation - Determination of the cause of failure was inconclusive.
The inspector witnessed numerous discussions between management, maintenance, engineering, vendor representatives, and consultants on
.the.cause of. failure. Evaluations for the exact cause of failure were inconclusive. Damage to the seal was so extensive that there was little evidence to help determine the cause.
In addition to the
. seal, pump and pump motor bearings were examined and all appeared to be well within specified tolerances. The licensee determined that the most probable cause of the failure was improper assembly or
. installation although other possible causes such as manufacturer's defect, foreign material, or extended operations at low system pressure could not be completely eliminated. The state of the seal upon disassembly was similar to one which failed at the Nine Mile Point plant in 1988. That failure was attributed to improper installation which allowed the seal faces to shift during system
' pressurization.
,
The licensee action plan for correction of the "B" RR pump seal problem was completed.
Investigations and evaluations for the cause of failure were inconclusive.
Investigations included an
'
examination of all RR pump motor and pump bearings as well as an examination of both the "A" and "B" loop seals. The "B" pump
,
i
- _ - __ - _.___ _____- - -
_ _. - - - - -
..
-
_ - - -
- ___- _
.
.
.
..
-
.
,
,
'
'
.
.
'
auxiliary / impeller and bearing were removed and there were no
indications of excessive wear or other problems.
Because the cause of failure could not definitely be determined, the-licensee planned.to provide for increased monitoring of the RR pumps, particularly the "B" pump during the plant startup. That monitoring was. to consist of additional engineering personnel on-shift to evaluate pump parameters; increased. observation of the installed instrumentation including 1the seal. pressures, seal temperatures' pump motor vibration, seal injection flow, and seal
,
leakage; and hand-held vibration reading when.the pump was energized, at 500 psig, and at normal operating pressure. The licensee was to keep the NRC informed of any anomalies.in RR pump seal behavior during the startup. These understandings were
' discussed in a meeting between the licensee and Regional Management on June 16, 1989.- Because of a lack of adequate communication of the understandings described above, the engineers and operating shift personnel monitoring the pump seals were not fully aware of expectations of the NRC. Licensee management's responsiveness to NRC concerns was inadequate in that they did not properly communicate those concerns to their subordinates.
During system pressurization after startup, the operators and engineering. personnel noted that the upper seal staging pressure for the."B" RR pump was not responding to the increasing system pressure.
The pressure remained steady at 13 psig as the same instrument on the
"A RR pump increased from 12 psig to 74 psig. This was during
.
about a four hour period during which lower seal staging pressure was increased from about 40 psig to about 190 psig. At about 6:30 a.m. on June 20, 1989, the operators began an investigation of the valve lineup for the upper seal staging pressure instrument. They discovered that the instrument isolation valve for the "B" RR pump
.was closed. The valve had apparently been left shut after restoring from a' flush after the post maintenance seal leakage test. The licensee determined that the Assistant Shift Supervisor who had ordered the flush restoration had failed to specify reopening the instrument isolation valve. Technical Specification 6.8.1.a required that written. procedures be established, implemented, and maintained covering ~ equipment control. Technical Procedure CPS No. 2800.04, " Generic Plush Procedure," Step 8.7.5 required that valve restoration lineups be conducted as specified by the Shift / Assistant Supervisor. The licensee normally implemented those requirements by the use of Administrative Procedure CPS No. 1052.01,
" Conduct of System Lineups." Step 8.3.2.2.b of that procedure required that a partial lineup be performed upon restoration from a system drain.
In this case the Assistant Shift Supervisor failed to specify that the instrument isolation valve be included on a partial system lineup.
Failure of the Assistant Shift Supervisor to specify that the instrument isolation valve for the "B" RR pump be properly restored after a system flush ia a violation of Technical Specification 6.8.1 (461/89021-01).
p
__
_ _ _ - _ _ _ _ - _ _ _
_ _
_
- - _ - _ _ _
P
,
..
.
.
g.
Incorporation of Vendor's Recommendation For Instrumentation and
-
l Operation.
The inspectors reviewed various procedures, references, and design
,/
documents to assess whether vendor recommendations for RR pump seal instrumentation, annunciation, and normal and abnormal operating procedures were incorporated at Clinton Power Station.
q W ing November 1985, Illinois Power Company's Nuclear Station
.
I
.
. gimering Department reviewed and verified the General Electric Company forwarded document K2801-0005 entitled, " Installation, Operation, and Maintenance Instructions - 20x20x33 Type RV Reactor Recirculation Pump Manufactured by Bingham-Willamette
.
'
Company...for...Clinton 1."
This manual provided the recommendations of the pump vendor for the installation, operation,
!
maint2rence, and repair of its RR pumps and the associated
. type 750 RV seal cartridges at Clinton. Sections 3 and 4 of the
'
.
manual contained the specific recommendations for instrumentation, i
annunciation, and operation. The inspectors noted several i
discrepancies between the vendor's recommendations and Clinton's installation and procedures. The inspectors requested that the licensee address the following concerns:
-
(1) Table 3-2 of the vendor's manual recommended upper seal staging pressure high/ low alarms at 810/270 psig. Clinton did not have alarms on that instrument.
(2) Table 3-2 recommended seal injection water temperature
,
in +rumentation with an alarm at 185 degrees Fahrenheit.
J11nton did not have the instrument or alarm.
~(3) Table 3-2 recomended upper seal staging #10w high/ low alarms at 1.3/0.6 gpm. Clinton's setpoints for ;taging flow alarms were 1.4/0.35 gpm.
(4) Paragraphs 3-5.2 and 4.7 of the manual recommended that the pump be operated no more than 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> with one of the parameters on Table 3-2 in the alarm condition. Clinton procedures did not contain those restrictions.
(5) Paragraphs 3-5.2 and 4-7 of the marnal also stated that the pump must be shutdown immediately if any of the shutdown limits on Table 3-2 were reached. Clinton procedures did not contain those requirements.
,
' #
(6) Paragraph 3-5.1.f of the manual recommended that the time of pump operation below 300 psig suction pressure be minimized.
Further clarification of that recommendation was provided by i
General Electric (GE) Service Information Letter (SIL) No. 203 dated October 29, 1976, and Supplement I to the SIL dated March 1980, in which GE recommended minimizing operation and
-
the number of starts of the RR pumps at suction
>
..
_ _ _ _ - _ _ _ _ _ - _ _ _ _ _ _ _ _ - _ _ - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ - _ _ _ _
_ _ - _ _ _ _ -.
.
'.
.
.
'
pressures below 300 psig to avoid shortening seal life.
Clinton Operating Procedure CPS No. 3302.01, " Reactor Recirculation," Precaution 4.1 str ted, " Minimize Recirc Pump starts below 300 psig reactor pressure to prolong seal life."
Limitation 6.4 of the same procec-re stated, "At low reactor pressure, Recirc Pump op:. etion 5 xuld be minimized to extend pump seal life. For exte..M. cages, Recirc Pumps may be shut down once Residual Heat Removal (RHR) is in operation." Thus, although Clinton procedures contained those precautions and limitations, no specific operator guidance was given. The recommendations were generally not considered limiting when scheduling plant operations. The "B" RR pump seal which failed had over 1000. hours of operating history at less than 300 psig system pressure. The "A" RR pump seal which was found to have no degradation had over 1200 hours0.0139 days <br />0.333 hours <br />0.00198 weeks <br />4.566e-4 months <br /> of low pressure oper: tion.
The licensee responded to the above concerns in memorandum Y-91658 dated June 15, 1989. The response was based on an evaluation of the system instrumentation and operating limits by GE dated June 14, 1989. The licensee stated that the vendor's recommendations were designed to protect specific equipment but that overall plant
,
demands and operating conditions as well as other information was used in establishing their incorporation. Leakage from the seals was bounded by the Technical Specification limits for unidentified reactor coolant system leakage and pump seal instrumentation was not required for any safety action. Thus, the licensee determined that the installed instrumentation and operatin) procedures were adequate.
6.
Failure of Feedwater Regulating Valve IFWOO4 The function of this valve was to regulate feedwater flow from the MORFP ("C" pump) during startup and low power operation. The Paul-Munroe operator was a self-contained, linear modulating, electro-hydraulic valve operator, which responds to a demand signal based on reactor pressure vessel (RPV) water level. Opening or closing of the valve was accomplished by energizing one of the solenoids to either retract or extend the operator cylinder rod. The design of the operator was such that a loss of demand signal should result in the actuator locking in the last position. During the failure on June 1,1989, however, the valve apparently locked in position without a loss of demand signal.
A review of the computer trace of plant parameters showed that from the time the "C" pump was started until the time of the scram, the feedwater demand signal was consistent with RPV water level. The "C" pump flow trace, however, indicated that the response of the flow regulating valve to the demand signal was intermittent. The intermittent response resulted in the valve locking in the full open position just prior to initiation of the manual scram.
o_
_
P[,
"
w
- ..
.
,
L
.
L
~
~
The NRC'-inspector reviewed the Maintenance Work Requests (MWRs) issued prior to this event-in an effort to establish the failure history of this valve.
,
i
- The following MWR's.Were'of-particular interest with respect to the e
current failure:
MWR No. C30482 was. issued on 3/9/87.to correct erratic position
-
L changes of the valve and failure of the valve to " lock" in position upon control signal failure. The servo-art /lifier board and solenoid valve assemblies were replaced to' effect. repairs. These components were. subsequently returned to the factory for failure analysis.
MWR No. C49953 was issued on 8/6/87 to correct. failure of valve to
-
respond to control signal changes. Upon inspection, low coil resistance was found on the solenoid valves, and-intermittent fuse
~ behavior -' causing loss of power to the servo-amplifier was noted.
~
MWR No. D14075 was issued on 5/26/89 to correct intermittent failure
-
of valve to respond to control signal. The servo-amplifier board
.was replaced.
From the history of valve control problems, it appeared that the servo-amplifier board may have been a weak link in the control system.
The licensee' developed a troubleshooting plan in an attempt to establish the root cause of the valve failure. The inspector observed various portions of the troubleshooting effort. The significant results of the licensee's effort are summarized as follows:
The pickup voltage for the installed solenoid valves was borderline
-
"and the coil resistance did not meet specification. These conditions may have contributed to overall valve failure however
.they were unlikely to be the root cause. Additionally, the pickup voltage for one of the spare solenoid valve assemblies was found to be unacceptable and was returned to the vendor.
Inspection of the control wiring and terminations identified an
-
intermittent contact symptom when the J1 connector on the servo-amplifier board was wiggled. While this condition could be repeatedly reproduced with the board installed in the junction box, it could not be duplicated on the bench. Loose electrical
'
connections may have contributed and, in fact, may have been the root cause of the valve failure.
It was hypothesized that vibration and temperature considerations may explain the intermittent nature of the problem. The servo-amplifier board from both the current failure and a previous failure that occurred on May 26, 1989, were returned to the vendor for failure analysis. Additionally, abnormal firing of the servo-amplifier silicone controlled rectifiers (SCRs)
was noted but analysis of the failed cards by the vendor and his review of the licensee's troubleshooting techniques indicated that the SCRs were damaged during troubleshooting.
i
-_
-
--
_ _ _ _ _
l
_ _ _
,i
-
.
'
Inspection of the actuator hydraulic system essentially ruled out a
-
contribution to failure.from this source.
Inspectibn of the feedwater regulating valve noted smooth operation
-
and no abnormalities.
The licensee elected to relocate the junction box which housed the servo-amplifier board from its previous position on the feedwater
. regulating-valve to a wall near the valve. This was done in an attempt to eliminate vibration of the control circuit which may have initiated the failure.
A new control board was purchased, tested to IP's specifications, installed, and functionally tested.
Upon completion of the dry run test, the valve wes functionally tested with feedwater operation. An action plan was followed during plant startup to monitor the valve's performance. No anomalies were noted.
7.
Trip of Drywell Chiller During the seal failure event, the on-service drywell chiller (VP-A)
tripped off in response to the increased heat load in the drywell.
Control room operators responded to the "VP-Trouble" alarm received in the control room by having an Area Operator start the standby VP-B chiller.
Drywell chiller-B was operating about 15 minutes after the initial indication of RR pump seal failure. The increased heat load in the drywell due to the pump seal failure increased the refrigerant pressure to a setpoint of 77 psig where a pressure transmitter signel actuated a pressure controller to open the chiller condenser outlei. valve IWS066A. During normal plant operation, Plant Service Water (WS)
provides the heat sink to the VP chillers at a pressure of about 110 psig. Since the VP-A Chiller had tripped on low condenser water pressure, the pressure transient reached the trip setpoint of 75 psig.
The licensee prepared a Troubleshooting Plan to verify the VP-A chiller trip was due to the WS pressure transient. The results of that effort were to be evaluated to determine what corrective action, if any, was to be implemented. The inspector reviewed the Troubleshooting Plan with cognizant licensee personnel and witnessed initial troubleshooting efforts.
a.
Troubleshooting Plan for Chiller Trip In order to analyze the transient loads placed on the drywell chillers and the affect those loads had on system performance, the licensee develogd a troubleshooting plan that would simulate load changes and allow observation of pressure changes in the systems.
The troubleshooting plan was approved by the Plant Manager and Engineering Manager prior to implementation.
As a prerequisite to the troubleshoot,ing effort, the low pressure service water trip circuit was Nimpered out and temporary test pressure gages were installed at the service water pressure switch vent instrument line. This arrangement allowed the service water
_ _ - _ _ _ _ _ _
-
= - - _ -
-
._
_ _.
_
__
b."
!
g
!
'
<
h a.
,
!
-
.
,
'
'
pressure to be monitored while troubleshooting without receipt of an
'
.
- actual ~ low service water trip signal. A simulated heat load was
'
.
i obtained by running the associated chill water pumps with their chiller secured.. Appropriate limitations on chill water temperatures were included in the troubleshooting plan.
With the prerequisites established, the A and B drywell chillers
were cycled on and off while recording pertinent transient data
"
including service water pressure, refrigerant pressure, and service water outlet valve (1WS066A/B) response.
'b.
Troubleshooting Results'
The above troubleshooting effort identified two anomalies in the "A"
-
drywell chiller. As had been suspected, the VP-A service water outlet pressure was observed to oscillate and " spike" below the chiller trip setpoint of 75 psig. An unexpected anomaly occurred when-the VP-A chiller. tripped several times on "avaporator low water
' temperature" when in fact a low evaporator water temperature was not present.
In addition to the two noted anomalies, the VP-A service water outlet valve, IWS066A, was observed to respond faster than the VP-B service water outlet valve. Throughout the troubleshooting effort, VP-B was observed to respond to transient loads without difficulty.-
The licensee's initial evaluation of the data collected through the
.
above troubleshooting effort concluded that the service water outlet pressure transient on VP-A was most likely due to the piping configuration and could be compensated for by installation of c pressure dampening device on the service water pressure instrument sensing line. The tripping of VP-A on evaporator low water temperature was due to a defective temperature sensing device.
c.
Corrective Action Maintenance Work Requests were initiated to install a pressure dampening device on VP chiller service water outlet pressure sensing lines and to replace the evaporator low temperature sensing device.
During replacement of the evaporator low temperature sensing device, the temperature sensor was found bent and positioned in contact with the chill-watnr-tube wall. The correct configuration was to have the temperature :ensor positioned in the center of its chill-water tube and held in place by support feet. The licensee concluded that with the ten,perature sensor in contact with the tube wall it was probably intermittently sensing the refrigerant
_(R-12) temperature causing the low temperature trips.
In addition, the licensee revised the low service water pressure trip setpoint from 75 psig down to 25 psig. The licensee stated that the purpose of the low service water trip signal was to assure service water availability and that by reducing the trip setpoint, a reduction in VP chiller trips due to service water pressure transients would result.
'
=
_ _ _ _
_ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ _
. _ _. _
_
_
-.
___
--
.
-- - -.
_-
-
.
'
i
}
.4
~
,"
d..
Equipment History The inspector reviewed previous Condition. Reports (CR) and Field Problem Reports (FPR) that had been initiated due to VP chiller problems.
As documented in FPR No~. 201,595, dated June.10, 1987, the licensee had identified the VP chiller trips due to service water transients
,
,
as unnecessary. The recommended resolution at that time (June 1987)
was to lower.the service water trip setpoint. Again, as documented in CRs No. 2-88-06-117 (dated June 23, 1988), and No. 2-88-10-161-(dated October 28,1988), the unnecessary VP chiller trips were identified by Plant staff.
In response to those CRs, a recommendation by the Nuclear Station Engineering Department (NSED)
to implement a design change and lower the servic:. water low pressure trip setpoint was to be proposed as a system improvement to the plant's Modification Review Committee. CR No. 2-89-04-098, dated April'20,1989,~ again identified VP chiller trips due to service water pressure transients; however, the NSED ren9nse to that CR was not due until June 20, 1989.
FPR No. 201,954 dated March 15,-1988, identified a problem with VP-A cycling on low chill water temperature. The disposition of.that FPR stated that the low chill water temperature cycling observed on VP-A was the normal operating sequence of.the VP chillers.
e.
Summary Based on the above, the inspector concluded that the licensee had performed an adequate investigation into the VP-A chiller trip that occurred on June 1, 1989. The review of historical data indicated the licensee was aware of the potential for VP chiller trips during service water transients. That awareness of system performance most likely prompted plant operators to quickly respond to the VP-A trip on June 1 by starting the standby VP-B chiller within about 10 minutes.
The corrective actions taken appeared to correct the anomalies identified during the troubleshooting effort. The delay in providing a permanent fix to the service water pressure transient that was a known problem since June 1987, appeared to be the priority assigned the system improvement recommendations from NSED.
The fact that operator actions could compensate for system transients, the VP chillers were not safety-related components, and the licensee considered the system improvement a low priority design change all contributed to delaying a permanent fix. The inspector could not determine if the corrective action taken above would have been performed without the management attention received following the VP-A. trip during the June 1 loss of recirc pump seal event.
l l
,
L
__
-_-
.
...
_ _ _ _ _ _ _, _.
-_ _ __ _ - _ _ - - _ _ - -. _ _. _ - _ - _ _ _ _ _ _ _ _ _ _.
_ _ _ _ _ _ _ _ _ - _ _ _
y
_ _
,
,
.j.
'
..-
~'
v
.
- c; n
.;~ 8. -.
Exit Interview
,
The inspectors met with licensee ' representatives (denoted in Paragraph 1)
throughout the inspection and at the conclusion of the inspection on'
-)
.
' June 23, 1989... The inspectors summarized the scope and findings of the-inspection activities. The licensee acknowledged' the inspection
. ?
-
findings. The inspectors also discussed the likely informational content of the inspection report with regard to documents or processes reviewed by the inspectors during the inspection. The licensee did not identify any documents / processes as proprietary.
.
,
1'
Y l'
l
_ - _ -
I