IR 05000321/1999010
| ML20210E177 | |
| Person / Time | |
|---|---|
| Site: | Hatch  |
| Issue date: | 07/20/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II) |
| To: | |
| Shared Package | |
| ML20210E165 | List: |
| References | |
| 50-321-99-10, 50-366-99-10, NUDOCS 9907280149 | |
| Download: ML20210E177 (24) | |
Text
.
.
U.S. NUCLEAR REGULATORY COMMISSION REGION ll Docket Nos: 50-321 and 50-366 License Nos: DPR-57 and NPF-5 Report Nos: 50-321/99-10 and 50-366/99-10 Licensee:
Southern Nuclear Operating Company, Inc. (SNC)
Facility:
E. I. Hatch Plant, Units 1 and 2 Location:
P. O. Box 2010 Baxley, Georgia 31515 Dates:
June 16 - June 25,1999 Inspectors:
B. Holbrook, Team Leader, Senior Project Engineer J. Munday, Senior Resident inspector T. Fredette, Resident inspector P. Fillion, Reactor inspector Approved by: P. Skinner, Chief, Reactor Projects Branch 2 Division of Reactor Projects
.-
Enclosure 1
!
9907290149 990720 DR ADOCK 0
1
.
EXECUTIVE SUMMARY Hatch Nuclear Plant, Units 1 & 2 NRC Inspection Report 50-321/99-10 and 50-366/99-10 This special team inspection included aspects of licensee operations, engineering, maintenance, problem identification and resolution, and corrective actions following a manual scram on Unit 2 due to a loss of condenser vacuum.
Operations Operations correctly identified and documented equipment and systems that were in a
.
Limiting Condition for Operation. Tracking files identified the equipment and applicable Technical Specification (TS). Although some minor deficiencies with respect to operator log documentation existed, the sequence of events was reconstructed from available log and safety parameter display system data (Section O3.1).
Operator actions and use of plant procedures were satisfactory. Licensee action to
.
clarify an unclear step of the Emergency Operating Procedures was appropriate (Section O4.1).
The evert review team conducted a thorough and detailed review of the eve.it. The
.
short term corrective actions to prevent recurrence were reasonable. Although the correctivo actions for a previous loss of condenser vacuum condition were reasonable, the licensee missed two opportunities to implement Event Review Team recommended corrective actions to improve early detection for this condition following a unit scram in 1995 and 1997 (Section O7.1).
j Maintenance The Temporary Modification used to implement a Residual Heat Removal Service Water
.
vent line leak repair, the associated 10 CFR 50.59 evaluation, and the maintenance activities were satisfactory. The licensee's corrective actions following previous and the most recent small bore pipe failures due to cycle fatigue was reasonable and satisfactory (Sections M2.1).
The increase in the total number of non-outage corrective maintenance work orders
.
contributed to untimely corrective maintenance of the "B" Control Rod Drive pump suction fiber. The non-outage corrective maintenance backlog for both units had more than doubled within the last year. Safety related c auipment was properly scheduled and repaired in accordance with plant procedures (M2.2).
Enoineerina The Main Steam isolation Valve (MSIV) solenoid failure that resulted in a failure of MSIV
.
2821-F022B to close on demand appeared to be an isolated case. The root cause analysis of the failed solenoid valve was thorough and detailed. The corrective actions to replace the failed solenoid valve cluster, inspect the remaining solenoid valve clusters, and for future replacement of the solenoid valve clusters at a staggered interval
r
and inspection of the removed valves, was reasonable. The analysis that determined that there was no common mode failure was reasonable (Section E2.1)
A Non-Cited Violation 50-366/99-10-02, Inadequate Corrective Action to identify and
.
Correct an Electrical Ground Fault on a 600 V Safety-related System, was identified.
The root cause analysis for the tripping of four 600 V safety-related power circuit breakers on Unit 1 which occurred at approximately the same time as the reactor trip on Unit 2 was thorough and supported restart of the Unit (Section E2.2).
The actions in response to a 4160 V bus transfer failure were immediate and
.
appropriate. The transfer timing relay was evaluated and replaced and the proposed calibration procedure upgrade was reasonable. There was a lack of guidance for identifying, trending, and evaluating causes of repetitive instrument drift, tolerance, and performance data. The significance of this relay in the autotransfer scheme was never identified or communicated to technicians performing calibration checks, nor was repeat drift (as-found) data evaluated. The licensee's assessment of plant procedures to improve the guidance was reasonable (Section E2.3).
,
.
Report Details introduction The NRC conducted a special inspection following a Unit 2 manual reactor trip due to a loss of condenser vacuum on June 15,1999. Subsequent to the trip several significant equipment problems were identified. The 2C and 2D Balance of Plant (BOP) electrical buses did not automatically transfer to the startup transformer as designed. This resulted in a loss of BOP equipment including condensate and feedwater flow to the reactor vessel. A primary containment isolation valve failed to close on demand as designed. An electrical ground that originated on Unit 2 caused unexpected tripping of Unit 1 equipment which was normally in service and also caused Engineered Safety Feature equipment to automatically start and align to their emergency configuration. During the event, a leak developed on the Residual Heat Removal Service Water (RHRSW) system which further challenged the recovery activities. The Resident inspector staff immediately reported to the site to assess the safety significance of the problems and licensee actions in response to the event. A team of regional based inspectors, including the resident inspectors reported to the site to assess the licensee's actions for problem identification and resolution, corrective actions, management oversight, and cverall performance associated with the event. A sequence of events is provided in Enclosure 2.
Event Synopsis On June 15, at approximately 8:10 p.m., Unit 2 control room operators identified a reduction in electrical generation, increased differential pressure and temperatures across the condenser, and a decrease in condenser vacuum. Reactor power was manually reduced in an effort to improve condenser vacuum, however, the decreasing trend was not reversed. At 9:24 p.m.,
operators manually tripped the reactor from about 30 percent power. Following the reactor trip and main generator trip, the 2C and 2D BOP electrical buses did not automatically transfer to the startup transformer as designed. Operators recognized this problem and manually aligned power to the buses. The failure of the BOP electrical buses to transfer caused a loss of both Reactor Recirculation (RR) pumps and a loss of condensate and feedwater flow to the reactor.
The operators manually initiated the Reactor Core Isolation Cooling (RCIC) system to control reactor water level and the Safety Relief Valves (SRVs) were manually cycled as required to control reactor pressure. The main condenser vacuum breakers were opened to prevent over pressurization of the condenser. Main Steam Isolation Valve (MSIV) 2B21-F0228 failed to close on low condenser vacuum. All other MSIVs closed as designed.
The on-shift operations supervision called additional off-shift operators to report to the site to aid the operating staff. The High Pressure Coolant injection (HPCI) system was placed in service for reactor pressure control and manual cycling of the SRVs was discontinued. Both loops of the Residual Heat Removal (RHR) and RHRSW system were placed in the torus cooling mode of operation due to the high heat load in the torus resulting from operation of HPCI, RCIC and SRVs which discharged steam into the torus. At approximately 11:00 p.m.,
while operators attempted to restart RR pump B, an electrical ground was experienced on the Unit 1600 Volt bus 1D. The electrical perturbation resulted in the tripping of several Unit 1 motor control center (MCC) breakers including the 1B Reactor Protection Syctem (RPS)
breaker. Due to the loss of RPS, Unit 1 received a half scram, group isolations, selected normally operating equipment tripped, and engineered safety feature equipment automatically started. Unit 1 operators responded to these actions and returned the equipment to the normal configuration.
_ - _ _ _ _ _ _ _ _ _
i l
l L
l At approximately 1:50 a.m. on June 16, the Unit 2 operators received a HIGH-HIGH-HIGH instrument sump level alarm in the torus area. A subsequent investigation determined that a
short section of small bore piping used to vent the RHRSW system had developed a leak. In an effort to control the leak, operators secured the A loop of torus cooling and RHRSW.
However, because the A and B loops of RHRSW are interconnected on its discharge piping, this action only reduced the leak rate. The leak was subsequently repaired and operators observed that the torus area instrument sump level alarms cleared at about 10:00 a.m.,
j approximately eight hours after the leak occurred. The reactor was depressurized and placed
{
in a cold shutdown condition at about 10:00 a.m. on June 17.
1. Operations
Operations Procedures and Documentation O3.1 Review of Limitina Condition for Operation (LCO) and Operator Loa Documentation (92901)(93702)
The inspectors concluded that operations personnel correctly identified and documented equipment and systems that were in a LCO. Tracking files identified the equipment and applicable Technical Specification (TS). The inspectors concluded that, although some minor deficiencies with respect to operator log documentation existed, the sequence of events was reconstructed from available log and safety parameter display system data.
Operator Knowledge and Performance 04.1 Operator Performance Followina Unit 2 Reactor Trio a.
Inspection Scope (71707) (92901) (93702)
The inspectors reviewed Emergency, Abnormal, and System operating procedures and assessed operators use of the procedures during the Unit 2 event. Operator performance and actions were reviewed to ensure procedure and regulatory requirements were met.
b.
Observations and Findinas One inspector arrived at the site soon after the reactor trip occurred. Upon arrival, the inspector observed that the operators were controlling reactor water level with the RCIC system and reactor pressure with SRVs. Both loops of RHR had already been placed in service to provide torus cooling. The inspector observed system configurations by control room indication and observed that the RCIC and RHR systems were being I
!
operated properly and within the established limits of the appropriate procedures. In addition, the inspector verified that annunciators important to safety were being addressed.
,
i The inspector observed that the operators quickly responded when the torus area j
instrument sump water level High-High-High alarm actuated. Control room operators dispatched plant operators to investigate the alarm. The Shift Supervisor (SS) correctly
r
.
entered the appropriate Emergency Operating Procedures (EOPs) for the sump level alarm. Plant operators reported that the A loop of the RHRSW which was being used for torus cooling was the cause of the high sump level. The SS directed that the loop be removed from service. Once the loop was secured, the operating crew ascertained that the leak had not fully subsided. After reviewing the appropriate plant drawing, the operators concluded that the remaining loop of RHRSW being used for torus cooling would also have to be shutdown to stop the leak. The SS, Superintendent On Shift (SOS), and the Operations Superintendent discussed whether or not to secure the remaining loop of torus cooling.
The inspectors observed that step SC/L-1 of the EOP was confusing to the operators.
The EOPs indicated that all systems discharging into the torus instrument sump area should be isolated unless the system was being used to assure adequate core cooling, shut down the reactor, suppress a fire, or vent primary containment irrespective of offsite release rates. Isolating the systems was contingent upon not being able to i
restore and mair,tain water level in the sump to below the Maximum Normal Operating Level (High-High-High level alarm). In this case, the operators were not able to restore
,
the sump water level to below the High-High-High level alarm condition. The operators concluded that since the water level was not backing up into the torus floor, that they were maintaining the water level and the RHRSW system discharging into the sump area was not required to be isolated.
The inspectors later reviewed the Boiling Water Reactor Owners Group Emergency Procedure Guidelines / Severe Accident Guidelines, Appendix B, and concluded that the RHRSW was not required to be isolated under these conditions. The Bases stated, in part, that Reactor Pressure Vessel Control and Primary Containment Control contingencies are given a higher priority than the objectives of Secondary Containment Control and that systems that must be operated to perform other steps in the EPGs are therefore not isolated in this step. The inspectors concluded that since the RHRSW was being operated to mitigate Primary Containment Control problems-(high torus water temperature), the operator action was appropriate.
The inspectors discussed with licensee management that the EOP step was not clear to the operations personnel. For immediate corrective actions, licensee management conducted a Beginning of Shift Training (BOST) session for Operations personnel and clarified the EOP step. Licensee management initiated a detailed review of the EOP step SC/L-1 and informed the inspectors that the wording of the EOP flow chart would be revised.
c.
Conclusion The inspectors concluded that the operator actions and use of plant procedures were satisfactory. Licensee action to clarify the SC/L-1 step of the Emergency Operating Procedures was appropriat.
05 Operation Training and Qualification 05.1 Operation of the Reactor Core Isolation Coolina (RCIC) System a.
Insoection Scope (929011 The Operators on shift that operated the RCIC system documented a deficiency with respect to the performance of the RCIC flow / speed controller. The inspectors reviewed the deficiency, discussed the observed system performance with the operators, discussed system operation with the responsible system engineer, and reviewed the past performance history of the RCIC controller.
b.
Observations and Findinas Following the manual initiation of the RCIC system for level control, operators observed that the RCIC speed indication was oscillating about 40 to 60 rpm when the controller was in automatic and set at less than 300 gallons per minute (gpm) flow. Rated flow for the system is 400 gpm. The controller operated properly at flows greater than 300 gpm.
The operators continued to operate the system in manual control to maintain reactor level and reported the operation when in automatic control on a maintenance work order (MWO).
The system engineer discussed the RCIC controller problem with the operators that had observed actual system performance in the control room. Thc system engineer had the
operators place the controller back in automatic and vary the flow setpoint to monitor i
system response. No deficiencies were observed. The licensee contacted a RCIC system consultant who reported to the site and assisted with the system performance review.
The inspectors reviewed the performance history of the RCIC controller and observed that a MWO was initiated on November 20,1997, following a Unit 2 reactor trip.
Operators observed and reported that the flow controller was extremely erratic while operating at 250 gpm flow. The resolution for this MWO was to calibrate the system instruments. The system operability surveillance was then satisfactorily performed. No erratic system performance was observed.
The inspectors also reviewed documentation from the system engineer to the Operations Manager following resolution of the 1997 MWO that stated,in part, that the system was operating properly and that operators needed to be trained on system operation to ensure the system remains stable. The inspectors discussed if operations
]
management had requested or provided additional or specific training on the RCIC
system. The inspectors found no indications that operations had requested any specific
)
or additional trcining based on this system engineer's memorandum. The inspectors were informed by licensed operators that they did not recall ever seeing similar RCIC system responses during plant simulator trainin r
Additionally, the inspectors discussed the RCIC system performance with the system engineer and the system consultant. The inspectors were informed that in the periods discussed above, the system was operating properly. The system engineer stated that the operators may not have left the controller in automatic long enough to allow the system to stabilize and that changes in reactor pressure and cycling of the minimum flow valve could affect system flow and stability. However, these individuals identified that the system could be calibrated for a slightly slower response time and the potential for erratic operation would be reduced. A calibration of system instruments and the controller and an operability surveiilance test was scheduled to be completed during the week of June 28. The inspectors identdied an Inspector Followup Item (IFI) to review the results of instrument calibration and operability testing. This item was identified as IFl 50-366/9910-01: Review of the Unit 2 Reactor Core Isolation Cooling System Instrument and Controller Calibration and Operability Surveillance Test.
c.
Conclusions Operations personnel did not receive specific training to ensure a clear understanding of system performance over a varied range of operating conditions as recommended by the system engineer after similar observations were made in November 1997. Further
!
review of this issue by the NRC will be performed during the Safety System Engineering Inspection scheduled for the week of August 23,1999.
07 Quality Assurance in Operations 07.1 Review of Event Root Cause Analysis (40500)
a.
The inspectors reviewed the root cause analysis report, discussed the root cause with Event Review Team (ERT) members, site management, engineering, and operations personnel. The inspectors reviewed the proposed corrective actions, instructions presented to operators, and actions taken to prevent similar occurrences.
b.
Observations and Findinas The day after the event, an ERT was established to determine the root cause of the loss of vacuum. The team concluded that the decrease in vacuum was the result of the I
condenser being unable to remove the existing heat load. Although the team was unable to state conclusively the specific conditions that caused this to occurred, possible explanations included air binding and/or steam / air blanketing of condenser tubes or a combination of various conditions.
The ERT stated that air binding could have resulted from operation with lower than normal circulating water fiume levels. Flume level was lowered to facilitate required chlorination of the system. Over a period of time with the lowered fiume levels, air could
.
be entrained by the circulating water system and would collect in the condenser waterboxes. This would result in a reduction of the heat removal capability of the condenser which would cause an increase in hotwell condensate temperature. It was suspected that saturation conditions occurred which would degrade the condenser performance and perpetuate the problem until the condenser became ineffective. A
r-
.
modification had been installed during the last outage to provide a continuous vent path from the waterboxes, however, evidence now indicates that the design was inadequate and may also have contributed to the problem.
'
Although the root cause was not conclusively determined, the ERT developed an interim operating strategy that the licensee considers would serve to avoid the conditions which were observed during this event. Those recommendations included removing one of the redundant circulating water inlet screens, maintaining higher pump suction pit levels, and taking actions to reduce unit power based on degrading conditions identified with the circulating water and condensate systems. The various parameters and values at which action should be taken were identified in the interim operating strategy. This strategy was presented to the operators as an Operating Order and discussed during the operators Beginning of Shift Training (BOST). Although Unit 1 has not experienced a similar problem, the ERT was tasked to develop similar guidance for that unit.
The inspectors conducted a review of an ERT report following a Unit 2 trip due to a loss of condenser vacuum in April 1997. The inspectors observed that many of the precurser indications were the same as those which occurred during the most recent event. In 1997 the ERT determined the root cause to be reduced heat rejection of the condenser initiated by air entrapment in the water boxes. Some of the ERT recommended corrective actions were as follows: 1) set the fiume level low alarm at 116.5 feet or relocate the level instrument to the condenser circulation pump pit; 2) only lower the fiume level to 117 feet when chlorinating the fiume; 3) automate the chlorination system; 4) conduct chlorination with a different chemical; 5) consider installing automatic fiume level control, and 6) consider installing continuous vents on the water boxes. A Design Change Request (DCR) was completed to install the continuous vent system for the condenser on both units. This activity was completed during the last refueling outage on the units. A DCR was deveioped to relocate the fiume level indicators to the pump suction pit. However installation was deferred to year 2000 due to manpower restraints. The inspectors observed that following the most recent trip the DCR for moving the level indication on Unit 2 was completed and the DCR for Unit 1 was being considered for installation in the near future. The inspectors reviewed the current flume level alarm setpoint calibration data sheet and observed that the fiume low level acceptance criteria was 113.4 to 114.6 feet. The inspectors observed that the setpoints were not changed as recommended by the ERT and the fiume level could be as much as three and a half feet lower than that recommended. A loss of condenser vacuum resulteg in a reactor trip occurred in 1995 was also reviewed by the inspectors and the conclusions reached by the ERT at that time was similar to the conclusions documented in the 1997 event.
The inspectors reviewed the results of a Condenser Circulating Water Pump performance test completed by corporate engineering in May 1997, following impeller replacements for the pumps. The data indicated that a previous test completed in October 1995, indicated a total flow of approximately 8,000 gpm less than the flow per the original flow performance curve. The pump flow test in 1997, after impeller replacement, indicated that the flow was 38,000 gpm greater than the original flow performance curve. The inspectors observed that the test was conducted with maximum flume level and water temperature of between 75-85 degrees. One of the recommendations following that test was to maintain the fiume level at maximum level to
r
ensure optimum performance. The inspectors observed that plant procedures to perform fiume chlorination directed the operators to maintain the flume level such that the high level and low level alarms were not actuated. The inspectors concluded that procedures allowed operators to lower the fiume level below the point where the licensee identified that air entrainment would occur.
The inspectors observed that, although the previous corrective actions completed were reasonable, the licensee missed two opportunities to fully implement ERT recommended -
corrective actions to improve early detection for a loss of condenser vacuum following the events in 1995 and 1997.
The inspectors reviewed Operating Order 00-05-0699S dated June 22,1999, that provided guidance for monitoring indications for early detection for the loss of condenser vacuum and discussed the instructions with operators. The operators demonstrated a clear understanding of the instructions. The Shift Technical Advisor (STA)
demonstrated the ability to locate and monitor the computer points identified in the instructions.
c.
Conclusion The inspectors concluded that the ERT conducted a thorough and detailed review of the j
event. The short term corrective actions to prevent recurrence were reasonable.
Although the previous corrective actions completed were reasonable, the licensee missed two opportunities to implement ERT recom.nended corrective actions to improve early detection for a loss of condenser vacuum following a unit scram in 1995 and 1997.
Il Me!ntenance M2 Maintenance and Material Condition of Facility and Equipment M2.1 Failure Of RHRSW Valve 2E11-FV003 Vent Line a.
Inspection Scope (40500) (92902)
The inspectors reviewed the maintenance activities associated with the repair of the RHRSW vent line, reviewed past performance history of this vent line and similar line configurations, and reviewed licensee long term corrective actions. The maintenance and repair activities were reviewed to ensure procedure and regulatory requirements were met, b.
Observations and Findinas Approximately 4.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> into the event and 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> after both RHR loops were placed in torus cooling, the line containing vent valve 2E11-FV003 cracked. This line is a 3/4 inch RHRSW vent line located on the discharge of the RHR heat exchanger. Approximately eight hours after the leak began, maintenance personnel were able cut the line, thread the pipe and install a threaded pipe cap on the broken pipe terminating the leak. The licensee completed the maintenance work as a Temporary Modification (TM) until a
8 permanent repair is designed and completed. The pipe cap TM was a code repair adequate for permanent use, i
The inspectors reviewed the TM used to implement the repair and the associated 10 CFR 50.59 evaluation. No unreviewed safety questions existed. The documentation for the TM was correct and in accordance with plant procedures. The inspectors concluded that the repair activity and safety evaluation were satisfactory.
The licensee determined the root cause of the failure to be high cycle fatigue caused by excessive vibration. The piping arrangement of the vent line was as follows: the vent line extended up from an 18 inch RHRSW pipe approximately 2 inches. A 90 degree elbow then turned the pipe where it extended for approximately another 21 inches. The vent valve'was located toward the end of the 21 inch pipe and weighed approximately 11 pounds. The licensee identified that the failure occurred near the horizontal portion of the elbow in the heat affected zone of the elbow weld. With the use of magnification, the licensee identified a weld arc strike on the outside of the pipe in the area of the failure which made that location favorable for fatigue crack initiation.
The inspectors conducted an independent review of maintenance history to determine if similar failures had occurred in the past. There were five small bore pipe failures attributed totally or in part to cycle fatigue since 1993. The following failures were identified:
in 1995, this same piping configuration containing valve 2E11-FV003, was found e
leaking water at the coupling located next to the 18 inch RHRSW pipe. The cause was determined to be cyclic fatigue and the corrective action was to replace the pipe and elbow.
in 1996, a 3/4 inch vent line for valve 2N21-F134 was found to be leaking where e
it connected to the 24 inch feedwater line. The cause was determined to be high cycle fatigue. The line was replaced using a different type weld. The licensee walked down the feedwater lines on both units and identified 12 other lines with similar configuration. The lines were inspected during subsequent refueling outages and no deficiencies were identified.
e in 1997, a leak was identified on a 3/4 inch vent line containing valve 1E11-F3017. The cause was determined to be a weld anomaly and high cycle fatigue.
The valve had been replaced during the previous outage and the length of pipe had been increased from 14 to 27 inches. The corrective action was to remove the existing pipe, shorten its length, and use a different type weld. In addition, Class 1 small bore pipe welds made during the 1996 Unit 1 refueling outage were inspected and no deficiencies were identified.
- Licensee records indicated that there was one small bore piping failure in 1993 and 1994. The corrective actions were to repair the leaks.
The licensee conducted a broadness review of systems to look for configurations that could be vulnerable to similar failure. Sixteen systems were initially considered and eleven were excluded from immediate consideration because these systems were i
I
)
r
.
normally operating with steady state vibration and the accumulation of fatigue cycles would most likely have already caused failure. Other non-safety related systems that were accessible to isolate any potentialleaks were also excluded. The following five
.
systems were reviewed in detail; HPCI, RCIC, RHRSW, RHR and Core Spray (CS).
f The inspectors noted that only accessible piping was included in the review. Vibration readings were taken for systems in service and the licensee planned to take vibration readings on other systems as they were placed in service. The vibration data was to be sent to corporate engineering for analysis. Most of these actions were completed prior to unit startup. The engineering evaluation of the vibration data identified that a drain line and valve 2E11 FD003 on the RHRSW system did not pass the vibration screening criteria. However, a visual inspection of the line did not identify any cracks. The licensee informed the inspectors that this problem would be repaired by September 1999.
The inspectors discussed with licensee personnel the observation that the line for valve 2E11-FV003 was in service since original construction, had failed in 1995, and failed again in 1999. The licensee presented no explanation for the duration between these failures. The inspectors reviewed the code required inspection following the pipe replacement in 1995 and observed that the completed inspection met code requirements. The licensee issued Deficiency Card 99-04889 following the most recent leak to review the TM and evaluate a permanent repair.
c.
Conclusion The Temporary Modification used to implement the Residual Heat Removal Service Water vent line leak repair, the associated 10 CFR 50.59 evaluation, and the maintenance repair activities were satisfactory. The licensee's corrective actions following previous and the most recent small bore pipe failures due to cycle fatigue were reasonable and satisfactory.
M2.2 Control Rod Drive (CRD) Pumo Suction Filter Problem and Maintenance Work Order (MWO) Backloa a.
Inspection Scoce (92902)
The inspectors reviewed the circumstances surrounding tripping of the CRD pumps on low suction pressure following the Unit 2 reactor trip. The inspectors reviewed the site procedure for pump motor start restrictions and maintenance history for the pump suction filters and general maintenance backlog.
b.
Observations and Findinas The inspectors observed that during the event the operators hr.d difficulty keeping the CRD pumps in operation. The pumps frequently tripped on low suction pressure due to clogging of the pump suction filters. The 2A pump tripped twice within 10 minutes and the 2B pump tripped twice within about two hours. During this event the CRD system was not necessary for reactor level control or to insert control rods, however, the system was used to cool the control rod drives. The system has two CRD pumps and two pump suction filters with one in service and one in standby. The pump suction is normally
.
r-
from the condensate system and after a loss of the condensate system the suction for the CRD pump automatically aligns to the condensate storage tank.
The inspectors reviewed the performance history of the pump suction filters and observed that MWO 29900407 was initiated for the "B" suction filter on March 1,1999.
Plant operators observed that the suction filter differential pressure gauge indicated 4.0 psid and the procedure acceptance criteria was a reading of less than 3.99 psid. The inspectors observed that the MWO was identified as a priority 3 which was defined as prompt scheduling, normally within seven days.
In this case, the MWO was not worked until June 10 when the differential pressure indicator was determined to be operating properly. The suction filter was cleaned on June 16, during the event, after the pump tripped several times on low suction pressure.
The inspectors discussed this observation with licensee management. The inspectors were informed that the water from the condensate storage tank is not as clean as condensate water and CRD pump suction filter plugging was not unexpected. The inspectors observed that the condensate storage tank on both units had been cleaned
)
within the past two years. The inspectors were informed that the cleaning was not as successful as desired. The licensee informed the inspectors that the Unit 1 CRD system did not demonstrate similar suction filter plugging problems as Unit 2.
The inspectors reviewed the MWO backlog for both units. The inspectors observed that
the non-outage corrective maintenance backlog had more than doubled within the last year. The total non-outage corrective MWOs for Unit 1 had increased from about 160 on November 15,1998, to 360 on May 15,1999, and had been reduced to 237 on June 15,1999. The non-outage corrective MWOs on Unit 2 had increased from about 80 on November 15,1998, to 225 on May 15,1999, and reduced to 161 on June 15,1999.
The inspectors were informed that actions were being taken to reduce the number of outstanding MWOs.
.
The inspectors discussed this observation with maintenance management.
Management stated that some reasons for the increased number of MWOs was: 1)
,
plant personnel had lowered the threshold for identifying deficiencies and more were
{
being identified; 2) the volume of maintenance work had increased since the fall of 1998 due to scheduled refueling outages and forced outages on both units; and 3) manpower
restraints. The inspectors questioned maintenance supervision about the work review and prioritization process to ensure safety related and equipment important to safety was being properly scheduled and worked. The inspectors were informed that work was being reviewed and properly scheduled. The inspectors did not identify any safety related equipment or additional equipment important to safety that was not prioritized and repaired within a reasonable amount of time.
i
c:
'
c.
Conclusion The inspectors concluded that the increase in the total number of non-outage corrective maintenance work orders contributed to untimely corrective maintenance of the "B" CRD pump suction filter. The inspectors observed that the non-outage corrective maintenance backlog for both units had more than doubled within the last year. The inspectors concluded that safety related equipment was properly scheduled and repaired in accordance with plant procedures.
111 Engineering E2 Engineering Support of Facilities and Equipment E2.1 Failure of Main Steam isolation Valve (MSIV) to Close on Demand a.
Inspection Scoce (92902) (92903) (40500)
The inspectors reviewed the circumstances surrounding the failure of a Unit 2 inboard MSIV to close on demand. The inspectors reviewed the licensee's actions associated with the root cause analysis, safety assessment, and corrective actions for the valve failure.
b.
Observations and Findinas As a result of the loss of condenser vacuum, the MSIVs received a close signal. All valves closed with the exception of 2B21-F022B. The operators attempt to close the valve from the main control room was not successful. When access was gained to the valve, an electrician tapped the DC solenoid and the MSIV closed.
The solenoid valve is part of a three valve cluster, containing an AC test solenoid and one AC and one DC solenoid used to operate the valve. The valve was model 975-30710 manufactured by Automatic Valve Company (AVCO). During operation, with the MSIV open, the latter two solenoids are continuously energized. Although either of these two solenoids will open the MSIV, both are required to de-energize for the valve to close. The licensee removed the solenoid valve cluster and sent it to a test facility for failure analycis. The valve was disassembled and the following was observed:
A small amount of black powdery residue was found around the plunger of the e
DC solenoid. It did not show signs of adhesion at room temperatures.
- The AC test solenoid did not show any signs of abnormal behavior or wear.
The AC operating solenoid was found to have the plunger bound at the lower e
end of the sleeve in which it moves. A significant amount of black residue was observed. Additionally, the dimensions of the plunger in both length and diameter were slightly greater than the dimensions of the plungers from the other two solenoids. The top portion of the plunger was described as a " flat topped mushroom" which increased its diameter in that location. The plungers for the
!
r
two AC solenoids were swapped and inserted in the other solenoid's sleeve.
The plunger for the test AC solenoid satisfactorily operated in the operating AC solenoid sleeve, while the operating AC solenoid plunger experienced binding in the test AC solenoid sleeve.
The valve manufacturer concluded that the failure was the result of "pcening" of the plunger. The peening occurs when a piece of debris prevents the plunger from traveling the full distance while in the energized state. The AC current putis the plunger upward opposing spring force, but, as the current amplitude decreased to "0", the spring forces the plunger to move slightly away from the " energized" position. As the coil reaches its peak current again, the plunger is pulled back to the energized position. This sets up a dynamic condition of the plunger moving based on the 60 Hz frequency of the AC soienoid. The black residue was described by the manufacturer to be stainless steel residue as a result of the rubbing between the plunger and the sleeve. It appeared in the DC solenoid because the air exhausted from the AC operating solenoid is vented through the DC solenoid and was not the result of friction in the DC solenoid. The manufacturer stated that failures of this type were highly uncommon. The licensee stated that this is the first failure of this type at Plant Hatch. However, the licensee had issued a 10 CFR Part 21 in 1993, when debris from the manufacturing process was detected in a different model solenoid valve from the same manufacturer.
The inspectors reviewed the root cause of the failure, the engineering safety assessment, and common mode failure analysis. The licensee's corrective action included replacing the faulty MSIV solenoid cluster. In addition, the remaining MSIV solenoid clusters were inspected for signs of " buzz or chatter", which the manufacturer stated would be observed if the peening condition existed. No deficiencies were observed. The licensee was also considering replacing the solenoid clusters at staggered intervals and inspecting the removed solenoids for signs of " peening" or contamination of the nitrogen supply to the valves. Additionally, engineering recommended that since foreign debris could initiate this type of failure, the Foreign Materials Exclusion procedures should be emphasized during removal or installation of the valves. The inspectors reviewed work packages and discussed the inspection with craftsmen who performed inspection for " buzz or chatter." The craftsmen demonstrated knowledge of the inspection techniques.
c.
Conclusion The inspectors concluded that the MSIV solenoid valve failure that resulted in a failure of MSIV 2B21-F022B to close on demand appeared to be an isolated case. The root cause analysis of the failed solenoid valve cluster was thorough and detailed. The corrective actions to replace the failed solenoid valve cluster, inspect the remaining solenoid valve clusters, and for future replacement of the solenoid valve clusters at a staggered interval and inspect the removed valves, was reasonable. The analysis that determined that there was no common mode failure was also reasonable.
1
I
'
)
.
E2.2 investiaation of Cause for Triopina of Various 600 V Circuit Breakers
!
a.
Insoection Scoce (92903)
The licensee reported that about the time of the Unit 2 reactor trip, four 600 V safety-related power circuit breakers on Unit 1 tripped. The inspectors reviewed the root cause
{
analysis and corrective actions and independently assessed the cause.
j
b.
Observations and Findinas The four circuit breakers that tripped were at Unit 1600 V switchgear 1D, and fed the following Unit 1 loads:
e Motor control center 1 R24-S010 on the intake structure l
Motor control center 1 R24-S031 for control building loads e
Vital battery charger 1R44-S001 e
Reactor protection system motor generator set B.
e A recent plant design change altered the source of power for Unit 2 motor control center (MCC) 2R24-S018B. The source of power was changed from an inverter to Unit 1 switchgear 1D. Similarly, MCC 2R24-S018A was fed from Unit 1 switchgear 1C.
Among the loads fed from Unit 2 motor control centers 2R24-S018A and B were the RR pump discharge valves. At about the same time that the circuit breakers on switchgear 1D tripped, the Unit 2 RR pump discharge valves were being cycled by operations personnel following the unit trip. Operators noticed that the ground indicating lamps for switchgear 1D indicated a ground while the Unit 2 discharge valves were being cycled.
Trouble shooting located a ground fault on the cable for RR pump discharge valve 2B31-F0318. The ground fault was in the drywell where the armored cable enters a junction box. A clamp designed to press on the armor was actually pressing on the insulation and had cut through the insulation.
The licensee's explanation for the tripping of the four circuit breakers was as follows:
The ground fault which only appeared on the system when the 2B31-F031B valve was j
energized caused a transient on the ungrounded system which propagated to all parts
'
of the system directly connected to bus 1D. The transient was the result of the line-to-
-
ground voltage shift that accompanies a ground fault on an ungrounded system. The 1D breaker trip devices interpreted the transient as a fault or overload and went to the trip condition. Three of the trip devices were model GE RMS-9 devices. The licensee i
stated that these digital devices were known to spuriously trip in a random fashion due i
to ground fault induced transients. The phenomenon had occurred before at the plant and at other plants, and is described in NRC Inspection Reports 50-321/98-08 and 99-02, and Information Notice 93-75," Spurious Tripping of Low-Voltage Power Circuit Breakers with GE RMS-9 Digital Trip Units." The trip device at the breaker for 1R24-S010 was a Micro-Versa Trip (MVT) model trip device. This device had no history of spuriously tripping for the type of transient described above. Therefore, there was a
_
.-
_
.
very plausible explanation for the tripping of the three breakers with GE RMS-9 trip devices, but the cause of tripping of the breaker with the MVT device remained under investigation.
Corrective actions which were accomplished before restart of the unit included repairing the cable termination, inspecting a sample of other cables to look for similar faulty terminations, and performing a dielectric test on the cable for discharge valve 2B31-F031B. The licensee made measurements of the voltage to ground at 600 V switchgears 1C,1D,2C and 2D to give additional positive confirmation that there was no ground fault on the safety-related 600 V system. These measurements indicated that there was no more than a 2.7 percent deviation from average in the voltages. The licensee checked the resistance of the 30 amp fuse at the starter for valve F031B. The inspectors concluded that these corrective actions were sufficient to support restart of the unit. The fact that the root cause of the tripping of the circuit breaker with the MVT trip device had not been determined at time of restart was acceptable for the following three reasons:
it did not seem plausible that there were additional undetected grounds on the e
system of a magnitude that would cause sufficient fault current to trip the breaker, especially since no individual load protection devices had tripped. The licensee checked breaker coordination at the 1D bus, e
The MVT devices had been in service since 1985 and there was no other known case of spurious tripping, which indicated that this was an isolated event.
An investigative team will continue to search 'for the root cause, with a definitive e
action plan.
I A longer term corrective action stated in the preliminary cause evaluation report was to replace the RMS-9 trip devices and, if necessary, the MVT devices. This effort had been in progress for some time. A replacement device has been identified, but additional work must be performed to qualify the device for safety related applications.
The inspectors reviewed the history of grounds on the 600 V system. The inspectors requested a summary of all deficiency cards written against the four safety-related 600 V buses since 1996. Five of the deficiency cards were for grounds on the system.
These grounds were identified by the control room operator through observation of the ground indicating lamps on the main control board. The deficiency cards are summarized in the following table:
Date Bus Comment 8/6/96 1D Indication cleared. Recorder connected.
Ground notlocated. Used Abnormal Operating Procedure for ground indication respons r-
11/9/96 1C Ground located at 1 A diesel generator jacket water heater.
1/8/98 1D Slight ground. Appeared to be related to moisture intrusion as it was raining at time.
Ground not located.
10/14/98 1C Ground appeared when 2B31-F031 A stroked. Ground found at armor cable clamp.
5/5/99 1D Ground occurred at same time as Unit 2 scram. Breaker for S031 tripped. Ground not located.
The October 14,1998, deficiency card is significant because it is essentially the same problem as found during trouble shooting the June 1999 event. The plant design change that modified the source of power to MCCs 2R24-S018A and S018B also installed new cables to RR pump valves 2B31-F031 A and 2831-F031B. Shortly after the modification was completed, the ground indicating lamps indicated a ground on bus 1C when valve 2831-F031 A was stroked. Trouble shooting found that the recently installed cable had a ground due to an improperly installed armor-lock cable clamp.
Plant personnel did not inspect the corresponding cable for the 2831-F031B valve. The inspectors concluded that the licensee missed an opportunity to identify a design deficiency that caused the breaker to trip during this event. Inspectors researched the design change package to ascertain whether the cables had been meggered or high potential tested as part of the installation process. Inspectors found that the new cables had been high potential tested after installation but before termination. The circuit was not insulation resistance tested after the terminations were made. As a result, the grounds were not detected at the time of initial installation.
The May 5,1999, deficiency card is significant because the ground indication at Unit 1 was received at the same time as a scram on Unit 2. The deficiency card states that the feeder breaker to MCC 1R24-S031 tripped at this time as well. This event was essentially the same as the June 1999 event except that only one circuit breaker tripped instead of four. The deficiency card was closed without the ground being located. The licensee did not establish any relationship between the trip of Unit 2 and the ground and breaker trip on Unit 1. The inspectors concluded that the tie between the two units, i.e.
cable from bus 1D to MCC S018B should have been the key to finding the ground, as it ultimately was in June 1999.
'
The existence of a ground on the safety-related 600 V ungrounded distribution system is a condition adverse to quality.10 CFR 50, Appendix B, Criterion XVI," Corrective Action" requires that conditions adverse to quality be promptly identified and corrected.
Two deficiency cards previously written by plant personnel did not locate the ground on the cable for valve 2831-F0318. This is a violation (VIO) of Criterion XVI. This licensee identified violation is being treated as a Non-Cited Violation (NCV) consistent with Appendix C of the NRC Enforcement Policy. This is identified as NCV 50-366/99-10-02, Inadequate Corrective Action to identify and Correct an Electrical Ground Fault on a 600 k
.
.
,
V Safety-related System. This NCV was in the licensee's corrective action program as DC 99-04885.
The inspectors were later informed that the armor-lock cable clamps iristalled to implement the repairs for the wiring ground deficiencies were the wrong size. The licensee was to conduct an engineering evaluation for suitability of the clamps. The licensee also informed the inspectors that the clamps would be replaced at the next available opportunity. The licensee placed this deficiency in the corrective action program as DC 99-05159.
The inspectors identified this problem as an Unresolved item (URI) pending the review of the suitability of the installed clamps, assessment of the engineering evaluation, and review of the circumstances of how the wrong size clamps were installed. This item was identified as URI 50-321,366/99-10-03: Review of Wrong Size Cable Clamp installation During the Ground Repair of the RR Pumps Discharge Valve.
c.
Conclusions The licensee's corrective actions to identify and correct an electrical ground fault were inadequate. This was identified as NCV 50-366/99-10-02, Inadequate Corrective Action to identify and Correct an Electrical Ground Fault on a 600 V Safety-related System.
The licensee's root cause analysis for the tripping of four 600 V safety-related power circuit breakers on Unit 1, which occurred about the same time as a trip on Unit 2, was thorough and supported restart of the unit.
E2.3 Failure of 4160 V Switchaear to Transfer Followina Unit 2 Reactor Trio a
Inspection Scope (37551) (40500) (93702)
The inspectors observed the licensee's investigation into the failure of the automatic i
transfer feature of the 2C and 2D 4160 V station service electrical buses following the Unit 2 reactor trip, and assessed the root cause and immediate corrective action to preclude this type of failure, b.
Observations and Findinas The inspectors reviewed the transfer scheme for the station service 4160 V buses 2A j
through 2D. The scheme was designed to automatically transfer (autotransfer) each bus from its unit auxiliary transformer (UAT) normal supply to the startup transformer
~ (SUT) alternate supply following any unit or main generator trip. Following the Unit 2 reactor scram, this autotransfer feature failed in that the alternate supply breaker never closed to maintain power to the station service loads, which included condensate and condensate booster pumps. As a result, operators experienced a loss of condensate j
and feedwater flow to the reactor in conjunction with the reactor trip.
'
The inspectors observed system engineers in the early stages of troubleshooting of the autotransfer failure. The root cause of the failure was attributed to a timing relay that had drifted out of tolerance, and contributing problems were identified in the opening
.
and closing times of the normal and alternate supply breakers. Part of the immediate corrective actions to conduct preventive maintenance of the normal supply breakers, with plans to accomplish the same maintenance on the alternate supply breakers at the next availability. The licensee included this problem in the corrective actions program as DC 99-04888. The licensee initiated Request for Engineering Assistance (REA) HT-97678 to review and revise the fast transfer timing aspects of the 4160 V transfer scheme. The inspectors determined that corrective actions relating to the breaker timing were satisfactory.
The inspectors followed up on the timing relay problem. The purpose of the relay was to ensure a " fast transfer" of the station service 4160 V buses to their alternate supply.
The timing relay is intended to abort closure of the alternate supply breaker to each bus after 10 60Hz cycles (~170 msec). The timing relay calibration was checked on June 16 and was found to be actuating too soon (~7-8 cycles) blocking the alternate supply breaker closing logic and preventing any autotransfer from taking place. This as-found setting was below the acceptance criteria of 9.810.2 cycles for this relay.
The inspectors reviewed the calibration history for this relay. The relay had just been placed in the licensee's calibration program in November 1998, and was initially calibrated at that time. Following the Unit 2 trip on May 5,1999, the system engineer requested the calibration of the relay be checked based on the suspicion that the autotransfer timing might be suspect. The system engineer made the request to two different technicians on two different occasions. As a result, the relay calibration was checked on two separate occasions; May 8 and May 10,1999. The inspectors reviewed the as-found and as-left data for each of these calibration checks. On May 8, the relay setting was found to have drifted to 17 cycles (70 percent out of tolerance). The technician recalibrated the relay to a setting of 10 cycles. On May 10, the second technician recorded the as-found data for the relay at 9.4 cycles, indicating that after 2 days, the relay setting had already drifted out of tolerance. The relay was recalibrated a second time. The as-found data for these two calibration activities and the drift problems was never communicated among the technicians or to the system engineer.
Not until the relay was found again out of tolerance on June 16 was a decision made to replace the relay. The licensee replaced and functionally checked the relay for
,
satisfactory timing, and planned to revise the calibration procedure to test the transfer scheme based on relay "normally closed" vice "normally open" contacts, which would be l
more conservative. The inspectors also verified that the calibration of the same timing relay on Unit 1 had been checked and was acceptable, i.e. no drift problems were evident.
I The inspectors questioned the system engineer and maintenance engineer regarding guidance fo tcthnicians in identifying relay and instrument drift problems, repeat problems Mth 'ay drift, and replacement criteria. The inspectors found that:
l
There is no guidance promulgated to technicians on identifying repetitive instrument drift problems or correlating as-found data as part of calibration activities. Essentially, as long as an instrument or relay can be calibrated, it is considered acceptable regardless of the as-found condition or setting.
l
r
,
i
An instrument is considered for replacement only if it either cannot be calibrated
- -
or is the cause of a plant problem and is evaluated for replacement by engineering.
)
There is no formal program for trending the as-found data from instrument e
,
calibration checks.
The inspectors determined that there was a lack of guidance for identifying, trending i
and evaluating causes of repetitive instrument drift, tolerance and performance data.
The timing relay for the 4160 V autotransfer scheme had been checked on three occasions since its inception in the calibration program in November 1998, and was found out of tolerance on each occasion. The significance of this relay in the autotransfer scheme was not identified or communicated to technicians performing calibration checks, nor was repeat drift (as-found) data evaluated. The inspectors were later informed that an assessment of plant procedures for improvements for the above observed deficiencies was initiated.
c.
Conclusions The inspectors concluded that the licensee's actions in response to the 4160 V bus transfer failure were immediate and appropriate. The transfer timing relay was evaluated and replaced and the proposed calibration procedure upgrade was reasonable. The inspectors determined that there was a lack of guidance for identifying, trending, and evaluating causes of repetitive instrument drift, tolerance, and performance data. The significance of this relay in the autotransfer scheme was not identified or communicated to technicians performing calibration checks, nor was repeat drift (as-found) data evaluated. The licensee's assessment of plant procedures to improve the guidance was reasonable.
V. Manaaement Meetinas and Other Areas X1 Exit Meeting Summary The inspectors presented the inspection results to members of licensee management on July 6,1999. The licensee acknowledged the findings presented.
The inspectors asked the licensee whether any material examined during the inspection should be considered proprietary. No proprietary information was identified.
Partiallist of oersons contacted i
Licensee Betsill, J., Assistant General Manager - Operations Crow, D., Licensing Manager Fomel, P., Performance Team Manager Fraser, O., Safety Audit and Engineering Review Supervisor Googe, M., Performance Team Manager Hammonds, J., Engineering Support Manager
F
.
.
Kirkley, W., Health Physics and Chemistry Manager Lewis, J., Training and Emergency Preparedness Manager s
Madison, D., Operations Manager Moore, C., Assistant General Manager - Plant Support Roberts, P., Outage and Planning Manager Sumner, L., Vive President, Hatch Tipps, S., Nuclear Safety and Compliance Manager Wells, P., General Manager - Nuclear Plant NRC Reyes, L., Regional Administrator, Region ll Plisco, L., Director, Division of Reactor Projects Skinner, P., Branch Chief, Division of Reactor Projects Holbrook, B., Senior Project Engineer Munday, J., Senior Resident inspector Fredette, T., Resident inspector inspection Procedures Used 37551:
Onsite Engineering 40500:
Effectiveness of Licensee Controls in identifying, Resolving, and Preventing Problems 62707:
Maintenance Observations 71707:
Plant Operations 92901:
Followup - Operations 92902:
Followup - Maintenance 92903:
Followup-Engineering 93702:
Prompt Onsite Response to Events at Operating Power Reactors items Opened. Closed. and Discussed j
Opened 50-366/99-10-01 IFl Review of the Unit 2 Reactor Core Isolation Cooling System Instrument and Controller Calibration and Operability Surveillance Test (Section 05.1).
50-366/99-10-03 URI Review of Wrong Size Cable Clamp Installation During the Ground Repair of the RR Pumps Discharge Valve (Section E2.2).
Closed 50-366/99-10-02 NCV Inadequate Corrective Action to identify and Correct An Electrical Ground Fault on a 600 V Safety-related System (Section E2.2).
r
!
'
.
.
Sequence of Events Hatch Unit 2 Loss of Condenser Vacuum and Subsequent Reactor Trip 06/15/99 20:10 Shift observed a reduction in electrical load and an increased differential in condenser circulating water (CCW) temperature and pressure with corresponding decrease in main condenser vacuum. Reactor power was manually reduced to approximately 65 percent.
06/15/99 20:25 Improving trend was observed on condenser vacuum and CCW temperature.
06/15/99 20:45 Operator stopped control rod insertion with Unit 2 at 42 percent power.
06/15/99 20:50 Operators verified that cooling tower fans were operating correctly. Cooling tower and CCW screen differential pressures were normal.
06/15/99 20:55 Control Rod insertion resumed.
06/15/99 21:08
" Turbine Vacuum Low" annunciator actuated.
06/15/99 21:10 Control rod insertion stopped with Unit 2 power at 29 percent.
06/15/99 21:24
" Low Condenser Vacuum" alarm received at 22.8" mercury (Hg).
Operators inserted manual reactor scram and entered Emergency Operating Procedures (EOPs). Balance of Plant (BOP) 4160 VAC buses 2C and 2D did not auto transfer to alternate source.
Operators entered EOPs to manually control reactor pressure with safety relief valves (SRVs).
06/15/99 21:25 Operators entered abnormal procedures for loss of 4160 VAC buses 2C and 2D, and loss of 600 VAC buses.
06/15/99 21:25 Operators manually initiated Reactor Core Isolation Cooling (RCIC) for level control.
06/15/99 21:28 Operators entered abnormal procedure for the loss of both Recirculation pumps.
06/15/99 21:38 Operators re-energized 2C 4160 VAC bus.
Enclosure 2
,
.
06/15/99 21:39 Operators re-energized 2D 4160 VAC bus.
06/15/99 21:54 2A Recirculation pump started 06/15/99 22:00 While attempting to start the 2B recirculation pump, operators received a ground on the 1D 600-volt bus and a loss of power to 1R24-S010, S031 and the vital AC Battery charger.
06/15/99 22:08 Unit 1B Reactor Protection System (RPS) motor generator set tripped, resulting in a half-scram and Group isolations. The trip was later identified to be due to the fault on the 1D 600-volt bus.
06/15/99 22:10 The ground was isolated, and the 2B Recirculation pump was started.
06/15/99 22:11 Unit 1 RPS B was manually transferred to alternate supply; the
,
half scram and Group isolations were confirmed and then reset.
06/15/99 22:21 Operators reported potential water flashing to steam in the main condenser due to high hotwell temperatures. Operators manually closed the outboard Main Steam isolation Vaives (MSIVs).
06/15/99 22:25 Operators manually opened the main condenser vacuum breakers.
06/15/99 22:26 Operators observed that the 2B inboard MSIV did not close on low condenser vacuum. Operators continued to maintain reactor pressure by operating SRVs.
06/15/99 22:40 Operators started the 2B Residual Heat Removal Service Water (RHRSW) pump in preparation for torus cooling.
06/15/99 22:42 Operators started the 2D RHRSW pump in preparation for torus cooling.
06/15/99 22:45 Operators started the 2A RHRSW pump in preparation for torus cooling.
06/15/99 22:46 Torus (suppression pool) temperature reached 100 F. Operators entered EOPs for primary and secondary containment control.
06/15/99 22:47 Operators started the 2C RHRSW pump in preparation for torus i
cooling.
06/15/99 22:50 Operators aligned the 2A loop of Residual Heat Removal (RHR)
for torus cooling mode in accordance with EOPs.
.
.
.
06/15/99 23:00 Operators started the 2A RHR pump for torus cooling.
06/15/99 23:02 Operators started the 2C RHR pump for torus cooling.
06/15/99 23:05 Operators started the 2B RHR pump for torus cooling.
06/15/99 23:07 Operators started the 2D RHR pump for torus cooling.
06/15/99 23:08 Operators verified all RHR and RHRSW pumps in service for torus cooling in accordance with EOPs.
06/15/99 23:57 Operators concluded a possible connection to the 1D 600-volt bus ground (22:00 entry) was due to opening the 2B recirculation pump discharge valve, which is fed from 2R24-S018B, which in i
turn is powered from the Unit 1D 600-volt bus.
'
06/16/99 00:00 Operators initiated plant cooldown in accordance with general operating procedures.
06/16/99 00:23 Operators manually started High Pressure Coolant injection (HPCI) for pressure control.
j 06/16/99 00:46 Operators observed suppression pool water level greater than 150" (primary containment EOP entry condition).
06/16/99 01:15 Operab 5 observed torus temperature greater than 110 F.
06/16/99 01:20 Torus water level was decreased less than 150".
06/16/99 01:39 Torus water levellowered to radwaste via RHR Loop B. Lowered from 155" to 148" (46583 gallons).
j 06/16/99 01:50 Received HIGH-HIGH-HIGH alarm annunciator on Torus Room Area instrument sump 2T45-N002A. It was determined that valve 2E11-FV003 on the RHRSW pipe in the torus has separated from the pipe. Operators entered the EOPs for leakage control.
06/16/99 01:55 Operators secured the 2A and 2C RHRSW pumps due to leakage at 2E11-FV003.
06/16/99 01:57 Operators secured the 2A loop of torus cooling in accordance with the EOPs to mitigate the leakage at 2E11-FV003.
06/16/99 03:01 Power Circuit Breaker PCB 179710 in the 500 kV switchyard trippe "
.
.
06/16/99 04:00 Operators observed torus temperature greater than 120*F, and initiated actions to place the unit in cold shutdown in accordance with plant Technical Specifications.
06/16/99 04:30 Torus water level still observed greater than 150". Unit was in Mode 3 and proceeding to cold shutdown.
06/16/99 05:43 The low suction pressure alarm for the 2A Control Rod Drive (CRD) pump actuated while operators were adjusting system i
flow. The CRD pump tripped and was immediately restarted.
06/16/99 05:44 The 2A CRD pump tripped again; operators initiated the abnormal operating procedure for loss of the CRD system.
06/16/99 05:49 The 2A CRD pump was returned to service; operators secured from the abnormal operating procedure for the CRD system.
06/16/99 05:53 The 2A CRD pump tripped again. Operators investigated the system lineup due to the 2B CRD suction filter being tagged out.
06/16/99 05:55 Operators transferred torus water to radwaste. Level was decreased from 157" to 154" (approximately 17,580 gallons transferred).
06/16/99 06:10 Operators started the 2B CRD pump with the suction filter crosstied and maintained flow as low as possible to prevent the pump from tripping on low suction pressure.
06/16/99 06:56 Operators re-opened the main steam lines (except for the B main steam line).
06/16/99 07:55 The 2B CRD pump tripped on low suction pressure.
06/16/99 07:58 Operators started the 2A CRD pump.
06/16/99 08:15 Plant operators started the mechanical vacuum pump to establish condenser vacuum. Excessive pump seal leakage was reported to maintenance.
06/16/99 08:17 Operators closed the main condenser vacuum breakers.
06/16/99 09:58 Operators observed that all torus area sump alarms had cleared and exited the EOPs associated with excessive leakage.
06/16/99 10:31 The 2A loop of RHR was re-established for torus cooling.
L_