ML20211Q579
| ML20211Q579 | |
| Person / Time | |
|---|---|
| Site: | Beaver Valley |
| Issue date: | 09/07/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20211Q563 | List: |
| References | |
| 50-412-99-07, NUDOCS 9909150098 | |
| Download: ML20211Q579 (29) | |
See also: IR 05000412/1999007
Text
,
.
~
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
)
Licensee No. NPF-73
Report No. 99-07
Docket No. 50-412
i
Licensee: Duquesne Light Company
Post Office Box 4
Shippingport, PA 15077
)
Facility: Beaver Valley Power Station, Unit 2
]
inspection Period: July 20 through 24,1999 - On Site
July 25 through 29,1999 - In Office Review
inspectors: Brian J. McDermott, Team Leader
Christopher G. Cahill, Reactor Engineer
Barry S. Norris, Senior Reactor Engineer
Geoffrey A. Wertz, Resident inspector
Approved by: Wayne D. Lanning, Director
Division of Reactor safety
l
!
l
9909150098 990907
PDR ADOCK 05000412
0 PDR ,
!
l
.
'
EXECUTIVE SUMMARY
A Special Team inspection was chartered to evaluate the operators' actions and equipment
J
response following a diesel generator failure at Beaver Valley (BV) Unit 2 on July 16,1999. The
Team also reviewed the circumstances that resulted in biofouling of the safety-related service
water system before, and after, the electrical event. (The team charter and an event timeline
are included as attachments to this inspection report).
On the basis of its independent review, the Team agreed with the licensee's root cause
determinations regarding the diesel generator failure and the failure of operators to take the -
expected actions in response to a loss of seal cooling for two reactor coolant pumps (RCPs).
Although Duquesne Light Company (DLCo) had not completed its own review of the service
water degradation at the conclusion of the on-site inspection, the Team agreed that adequate
short term corrective actions had been taken to support restart of the plant.
The major findings and conclusions of this inspection were:
Operations
The BV Unit 2 operating crew failed to recognize that all seal cooling for two RCPs was lost and
consequently, they did not implement actions specified in an alarm response procedure to
. protect the seals. The importance of these actions was not emphasized in training or indicated
by the human factoring of the control room annunciators. The failure to implement this
procedure is a violation which has been entered in the licensee's corrective action program and
is being treated as a non-cited violation consistent with the NRC Enforcement Policy. The
operating crew did, however, respond well to a loss of one 4 kV emergency bus and effectively
mitigated potential equipment problems. (Section O1.2)
The licensee's 1997 Probability Risk Assessment shows that reactor coolant pump seal failures
contribute 50% of the total core damage frequency for BV Unit 2. This risk insight was not
previously used to identify improvements in plant procedures, operator training, or control room
alarm human-factors that would assist with mitigation of this risk significant event. (Section
01.2)
The licensee failed to develop procedures for loss of emergency power, as required by
Regulatory Guide 1.33 and the Technical Specifications. Although Operations department
personnel knew procedure guidance was lacking in this area, the " loss of bus" procedure had
not been identified as a required procedure. This violation has been entered in the licensee's
corrective action program and is being treated as a non-cited violation consistent with the NRC
Enforcement Policy. _ (Section 03.1)
Maintenance
Macro biological fouling (biofouling) in the service water piping that supplies the diesel generator
was not detected during a biocide treatment on July 7. Seven days later, a rapid and substantial
- - degradation of service water flow occurred during an unrelated diesel generator surveillance
' test. The management team was slow to understand the effects of the July 7 biocide treatment
ii
.
I
'
- for common mode failure and failed to protect EDG 2-1 from heat exchanger fouling. This series
of events resulted in Emergency Diesel Generator 2-2 being inoperable for longer than the 72-
hour allowed outage time of the Technical Specifications. Additionally, the initial plan for EDG 2-
2 operability restoration was incomplete, until challenged by the inspectors. This apparent
violation of Technical Speedications is being considered for escalated enforcement action.
(Section M2.1)
In 1995, the licensee developed a plan for the prevention of biofouling in the service water
system. Although plans for the type of biocide treatments were established, frequencies for
those treatments were not included in the plan. Subsequently, the licensee failed to perform
these treatments consistently and frequently enough to be effective. An increase in the Zebra
mussel population at the service water intake structure in 1998 was a missed opportunity to
identify this problem. This apparent violation of Quality Assurance requirements for Corrective
Action is being considered for escalated enforcement action. (Section M2.2)
The licensee failed to provide adequate acceptance criteria in its procedure for bulk chemical
treatments of the service water system. Specifically, the emergency diesel generators were not
monitored to assess the impact of biofouling dislodged during the treatment. The lack of
acceptance criteria, coupled with the simultaneous treatment of both service water trains,
created the potential for a common mode failure and a significant reduction in safety margins.
The failure to develop an adequate procedure for bulk chemical treatments is an apparent
violation of Quality Assurance requirements and is being considered for escalated enforcement
action. (Section M2.3)
On July 7, a chemistry technician failed to follow the procedure for sampling of the service water
system during a bulk chemical treatment activity. As a result, the intended biocide concentration
was not applied to the "A" train of service water. This failure to follow procedures is being
treated as a Non-Cited Violation, consistent with Appendix C of the NRC Enforcement Policy.
(Section M2.3)
Enoineerina )
The licensee identified the most probable cause of the EDG 2-2 failure on July 16 as an
intermittent control relay contact in the voltage regulator circuit. A questioning attitude
throughout the engineering evaluation and root cause analysis was observed, and the licensee
made effective use of the vendor support. The voltage regulator repairs and retest were
appropriate. Long term corrective actions recommended by the Event Response Team are
reasonable actions to prevent recurrence of the relay failure. (Section E1.1)
Equipment aligned to the 4 kV emergency bus that was affected by the EDG 2-2 failure
responded as expected during the low voltage condition which existed for approximately 75
seconds. The licensee's engineering evaluations for loads that were running, or received start
signals, were technically sound. In addition, electrical tests were used to confirm the condition
of the affected equipment. (Section E1.2)
The licensee conducted adequate troubleshooting of the EDG 2-2 local control switch anomaly
that resulted in an unexpected start. Intermittent contact resistance during operation of a local
iii
.
-
~
'
control switch was identified as the most probable cause, after other possibilities were
eliminated through testing and design reviews. Appropriate short term actions were completed
and, long term corrective actions will be addressed by the licensee's corrective action process.
(Section E1.3)
The safety-related 125 voit batteries and DC system responded normally during the loss of
power to their chargers. TS requirements for the DC distribution system were appropriately
implemented. The licensee's engineers provided a reasonable assessment of the batteries'
performance during this event and concluded that the battery discharge rates were within their
design. An inspector followup item was opened to review the DC system's capability during a
loss of battery charging event that requires a plant shutdown. (Section E1.4)
The licensee appropriately evaluated the operational condition of the reactor coolant pump seals
after the loss of all seal cooling (concurrent loss of both seal injection and thermal barrier flow).
Based on plant data taken before, during, and after the event, the licensee determined no
significant heat up of the seals occurred. (Section E1.5)
r -
iv
E 1
.
TABLE OF CONTENTS
EXEC UTIVE SU MMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i1
Summary of Plant Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1, Operat ions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
01 Conduct of Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
01.1 General Comments on the Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
01.2 Operators' Response to the Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
03 Operations Procedures and Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
03.1 Lack of a Procedure for the Loss of an Emergency Bus. . . . . . . . . . . . 4
11. M airite n a nce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
M2 Maintenance and Material Condition of Facilities and Equipment. . ..... .... 5
M2.1 EDG 2-2 Service Water Flow Degradation . . . . . . . . . . . . . . . . . . . . . . 5
M.2.2 Program for Prevention of Macro Biological Fouling . . . . . . . . . . . . . . 7
M2.3 Service Water Chemical Treatment Procedure . . . . . . . . . . . . . . . . . . . 9
ll!. Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1
E1 Conduct of Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 11
E1.1 Root Cause Determination for EDG 2-2 Failure . . . . . . . . . . . . . . . , . 11
E1.2 Effects of Under Voltage on Emergency Bus 2DF . . . . . . . . . . . . . . 12
E1.3 ~ Unexpected Restart of EDG 2-2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
E1.4 - Loss of Charging for Battery 2-2 and Battery 2-4 . . . . . . . . . . . . . . . . 14 .
E1.5 Loss of Seal Cooling for Two Reactor Coolant Pumps . . . . . . . . . . . . 16
- V. Management M eetings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
X1 Exit Meeting Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
ITEMS OPENED, CLOSED, AND DISCUSSED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
q
t
LIST OF ACRONYMS USED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
l
I
ATTACHMENT A - SPECIAL INSPECTION TEAM CHARTER '
ATTACHMENT B - EVENT TIMELINE
l
i
i
V ;
,
._
.
Report Details
Summary of Plant Status
'
On July 14 at Beaver Valley (BV) Unit 2, the service water cooling flow to Emergency Diesel
Generator (EDG) 2-2 degraded to less than its design basis flow during a surveillance test. On
July 16, EDG 2-2 was started and synchronized to the grid through emergency bus 2DF for a
post maintenance test following the cleaning of its heat exchanger. During this test, a diesel
generator failure occurred which resulted in a loss of power to emergency bus 2DF and its
associated loads. The plant was shut down on July 18 after it became apparent that EDG 2-2
could not be declared operable within the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> period allowed by the Technical l
Specifications.
After completing a review of the equipment and operator issues associated with this event, i
Duquesne Light Company (DLCo) management concluded that the appropriate short term
corrective actions had been taken to support restart of the plant. On July 26, operators
commenced a reactor startup sequence for BV Unit 2.
1. Operations ,
l
'
01 Conduct of Operations'
01.1 General Comments on the Event
NRC Region I management chartered a Special Team inspection to review tha l
'
circumstances surrounding the EDG failure of July 16. In addition, the Team was
chartered to review the biofouling discovered in the service water system on July 14 and
July 21. The Team's charter and a sequence of events are included with this report as
Attachments A and B, respectively. ,
An initial NRC assessment determined that the risk significance for a 2-hour loss of one j
4 kV emergency bus was a small reduction in safety margin. Operators restored RCP
seal cooling in less than three minutes and significantly decreased the associated risk of l
an RCP seal loss-of-coolant-accident. Likewise, EDG 2-2 being inoperable for l
approximately 5 days was determined to be of small risk significance. During the 5 day l
'
period, the redundant EDG, off-site power, and a 4 kV cross-tie to BV Unit 1 were
available. This risk assessment, in part, was the basis for initiating the Special Team
inspection.
Based on the results of this inspection, the Team noted that the failure to implement an
adequate biofouling treatment program cculd have resulted in a common mode failure of
both EDGs, for a period of up to 14 days prior to detection. In this scenario only the off-
site power and the 4 kV cross-tie would have been available. An NRC assessment of
this scenario determined that it would result in a significant reduction in safety margins.
.
' Topical headings such as Oi, M8, etc., are used in accordance with the NRC standardized ,
reactor inspecten report outline. Individual reports are not expected to address all outline topics.
-. _ _ _ _ _ _ _ _ - _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.
.
2
01.2 Ooerators' Response to the Event
a. Inspection Scope (93702)
The inspectors reviewed the operator recovery actions associated with the loss of one
4kV emergency bus. The assessment included reviews of the licensee's Event
Response Team (ERT) report, the operator logs, the sequence of events recorder, and
the station procedures. Additionally, interviews were conducted with the operators,
training instructors, the ERT members, and senior station management.
b. Observations and Findinos
On July 16,1999, BV2 was at 100% power, conducting a full load surveillance test of the
2-2 emergency diesel generator (EDG). At 5:29 pm, one of the tie breakers (2F7)
between the 4 kV emergency bus (2DF) and normal 4kV bus opened. The operators
subsequently observed low voltage on the 4 kV emergency bus and opened the EDG's
output breaker (2F10). This de-energized the 4 kV emergency bus and the "B" train of
emergency equipment. Of immediate concem were the loss of Battery Chargers 2-2 and
2-4 (requiring a TS 3.0.3 required shutdown), and the loss of the "B" charging pump for
inventory control and reactor coolant pump seal injection.
The crew correctly diagnosed the plant conditions and the Assistant Nuclear Shift
Supervisor (ANSS) prioritized the operators' actions. Specifically, the plant operator
(PO) was directed to attempt to restore electrical power to the 4 kV emergency bus, and
the reactor operator (RO) was told to restore charging. The "A" charging pump was
manually started and seal injection was restored within 2 minutes and 45 seconds. After
verifying that no faults were indicated on 4 kV emergency bus 2DF, operators restored
its normal power supply (approximately two hours later).
The ANSS recognized that there was no abnormal operating procedure (AOP) to
address the loss of s 4 kV emergency bus. After the plant was in a stable condition, the
ANSS directed the operators to review the control panels and to ensure that all of the
alarms were expected. During this event approximately 120 annunciators had alarmed.
The RO subsequently identified that cooling flow to the thermal barrier heat exchanger
had been lost for the "B" and "C" RCPs when their attemating current (AC) powered
supply valves were de-energized. The ANSS then told the Shift Technical Advisor to
review the alarm response procedure (ARP) for loss of seal injection. This ARP required
operators to immediately trip the reactor, and to secure the affected RCPs within 2
minutes. The ANSS informed the NSS of the missed actions. Based on the fact that
seal injection was recovered within three minutes, and following an initial review of RCP
parameters, the ANSS (with NSS concurrence) considered the RCPs operable and
concluded that the actions in the ARP were no longer required. The ANSS then
requested a formal engineering evaluation of the seals.
During NRC interviews, the operators stated that they were not aware of the potential for
loss of all seal cooling to certain RCPs, with the loss of a single 4 kV bus. Also during
interviews, the ANSS told the inspectors he knew that a concurrent loss of seal injection
.-_ ____ .___ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ .
.
l
.
3
and thermal barrier flow required an immediate reactor trip. The inspectors concluded
that the failure to recognize that all seal coding was lost (and implement the required
immediate actions for protection of RCP seals) is significant and indicative of a weakness
in the training program.
Notwithstanding the condition of the pump seals, the failure to implement the actions
contained in ARP A2-4D, " Reactor Coolant Pump Seal Trouble," for a concurrent loss of
seal injection flow and thermal barrier cooling flow is a violation of the Beaver Valley,
Unit 2 TS, Section 6.8.1, requiring the implementation of procedures. This Severity Level
IV violation is being treated as a Non-Cited Violation (NCV), consistent with Appendix C
of the NRC Enforcement Policy. This violation is in the licensee's torrective action
program as CR 991752. (NCV 50-412/99-07-01)
The licensee determined the root causes for the operators failing to reference the ARP
were: (1) there was no mechanism to aid in identifying alarms that require prompt
operator actions, and (2) the operation's standard for response to a multi-alarm event did
not address prioritizing ARPs. Corrective actions included: (1) development of a
prioritization scheme for the annunciators, and (2) training on the new scheme with an
emphasis on the need to ensure that all annunciators are understood during an event.
These corrective actions were applicable to both Units.
The inspectors compared the actual plant response to that observed in the plant specific
simulator and noted no significant differences. After review of the ERT's final report
(issued July 24,-1999) and independent reviews, the inspectors concluded that the I
licensee had generally performed a good root cause evaluation. However, the
inspectors also considered the lack of training on the required, immediate actions to
protect RCP seals, and prevent an RCP seal LOCA, a significant problem.
The BV Unit 2 Probability Risk Assessment (PRA) Update Report, dated October 31,
1997, discussed the fact that a loss of both sources of cooling to the reactor coolant
pumps (RCPs) subjects the pump seals to full reactor coolant system temperatures. The
elevated temperatures cause a degradation of the RCP seals and would eventually lead
to a small break loss-of-coolant-accident (typically referred to as an RCP seal LOCA).
According to the licensee's PRA report, RCP seal LOCAs contribute 50% of the total
core damage frequency (CDF) for Unit 2. The inspectors determined that the licensee
did not effectively use this risk insight to improve the timeliness and reliability of
mitigating operator actions. Specifically this information had not previously been used to
improve procedures, training, or control room alarm human-factors.
c. Conclusion
The BV Unit 2 operat;ng crew failed to recognize that all seal cooling for two RCPs was
lost and consequently, they did not implement actions specified in an alarm response
procedure to protect the seals. The importance of these actions was not emphasized in
training or indicated by the human factoring of the control room annunciators. The failure
to implement this procedure is a violation which has been entered in the licensee's
corrective action program and is being treated as a non-cited violation consistent with the
_ _ _ _ _ _ _ .
i
i
4
violation consistent with the NRC Enforcement Policy. The operating crew did, however,
respond well to a loss of one 4 kV emergency bus and effectively mitigated potential
equipment problems.
The licensee's 1997 Probability Risk Assessment shows that reactor coolant pump seal
failures contribute 50% of the total core damage frequency for BV Unit 2. This risk
insight was not previously used to identify improvements in plant procedures, operator
training, or control room alarm human-factors that would assist with mitigation of this risk
significant event.
03 Operations Procedures and Documentation
O3.1 Lack of a Procedure for the Loss of an Emeroency Bus
a inspection Scope (93702)
The inspectors determined that neither BV Unit had procedures for a loss of an
emergency electrical bus. The inspectors discussed the issue with station management,
assessed the immediate corrective actions, and performed an independent assessment
to determine if other procedures were required.
b. Observations and Findinos
As noted in Section O1.2 of this report, the ANSS knew there was no procedure to
address the loss of a 4 kV emergency bus. Through interviews, the inspectors lesmed
that operators and operations managers knew of this deficiency. The drafting of a " loss
of bus procedure" had been initiated several years ago, but was never finished.
NRC Regulatory Guide 1.33 (RG1.33), " Quality Assurance Program Requirements
(Operation)," lists typical procedures used at nuclear facilities, including loss of electrical
power. Technical Specification requirements for both BV Units require procedures to be
developed for activities covered in RG 1.33. After the inspectors discussed this with
licensee management, a CR was initiated. The licensee had not previously recognized ,
the lack of this procedure as a failure to meet their TSs. l
For corrective action, the licensee developed four AOPs for each Unit and trained all the
crews prior to them assuming the shift. The new procedures cover: Loss of 4kV
Emergency Bus, Loss of 480V Bus, Loss of Vital Bus, and Loss of 125VDC Bus.
The inspectors observed portions of the simulator development and validation of the )
AOPs, reviewed the final procedures and the associated training package, and
discussed the development of the procedures with plant management. The Plant
Manager indicated that further consideration will be given to adding more detail to these
initial procedure revisions based on future bench-marking with other utilities. The
inspectors considered the new procedures, training, and plans for bench-marking to be
acceptable.
~
5
The inspectors also asked whether any other procedures required by RG 1.33, Appendix
"A", Section 6, were missing from either BV Unit. The licensee reviewed existing
procedures against the RG 1.33 list and concluded that no other procedures were
missing. The inspectors performed an independent review and no discrepancies were
identified.
Technical Specification Section 6.8.1, for both BV Units, requires written procedures to
be established covering the activities in Appendix "A" of RG1.33, Revision 2, February
1678. The failure to develop procedures required by Section 6.c of Appendix "A" of
RG1.33 is a violation of TS 6.8.1 for both Units. This Severity Level IV violation is being
treated as a Non-Cited Violation (NCV), consistent with Appendix C of the NRC
Enforcement Policy. This violation is in the licensee's corrective action program as
CR 991764. (NCV 50-334/99-07-02 & NCV 50-412/99-07-02)
c. Conclusion
The licensee failed to develop procedures for loss of emergency power, as required by
Regulatory Guide 1.33 and the Technical Specifications. Although Operations
department personnel knew procedure guidance was lacking in this area, the " loss of
bus" procedure had not been identified as a required procedure. This violation has been
entered in the licensee's corrective action program and is being treated as a non-cited
violation consistent with the NRC Enforcement Policy.
11. Maintenance
M2 Maintenance and Material Condition of Facilities and Equipment
M2.1 EDG 2-2 Service Water Flow Deoradation
a. Inspection Scope (62707,93702)
During a surveillance test for EDG 2-2 on July 14, operators found that the SW flow to
the EDG had decreased from about 1500 gpm to below the design basis minimum flow
of 1170 gpm after fifteen minutes. The EDG was declared inoperable and a
maintenance inspection discovered about 3 gallons of biological fouling (primarily Zebra
mussels) blocking approximately 90% the heat exchanger's tube sheet. The inspectors
reviewed the details of the event, the duration of the degraded condition, and the safety
implications associated with the cause of the degradation.
b. Observations and Findinos
On July 7, the licensee performed a bulk chemical treatment of the service water system
in accordance with procedure 2-OM-30.4.M, Revision 7, the treatment program for all
biofouling. On July 14, the initial flow of SW to EDG 2-2 during a surveillance test
resulted in the rapid accumulation of Zebra mussels in the EDG's heat exchanger. The
minimum SW flow to the EDG heat exchangers specified in the Updated Final Safety
.
6
approximately 1070 gpm, prior to the operators stopping the test. EDG 2-2 was declared
inoperable on July 14 at 9:57 p.m. Sections M2.2 and M2.3 of this report provide
additional detail regarding the causes of this event.
Station management met on July 15, after cleaning the EDG 2-2 heat exchangers, to
develop a plan to restore EDG 2-2 to operable status. At the conclusion of this meeting, l
the approved post maintenance test required SW flow through EDG 2-2 for 30 minutes.
If SW flow remained above 1300 gpm at the end of the test, EDG 2-2 would be declared
operable. In addition, SW flow would be established through EDG 2-1 for 30 minutes per
day for one week to evaluate the potential for a common mode failure. The inspectors
determined that the EDG 2-2 restoration plan and EDG 2-1 monitoring plan were
incomplete. The acceptance criteria permitted flow degradation from the normal 1550
gpm to 1300 gpm in 30 minutes to be acceptable for EDG 2-2 operability. These plans
did not address the 7 day ECG mission time as described in the Updated Final
Safety Analysis Report (UFSAR). The acceptance criteria did not consider
appropriate heat exchanger flow characteristics. Specifically, SW flow would
appear normal as the first 20-40% of the tube sheet became fouled, but drop off
more rapidly as additional Zebra mussel shells accumulated to block the remaining
portion of the tube sheet. Station personnel had not identified the source of the
macro biological fouling and did not address the potential for further biological
fouling accumulation on various heat exchangers. The inspectors discussed these ,
issues with station management. The licensee agreed that the EDG restoration and
monitoring plans had been incomplete and required revision.
The revised SW flow test acceptance criteria did not include the effect of river
water level or SW system pressure on the minimal SW flow to the EDG heat
exchangers. When questioned by the NRC inspectors, the system engineer
developed a curve of system flow versus pressure which addressed these
parameters. However, this curve was not provided to operators until three days
after the monitoring period began. The inspectors determined that not having timely
and accurate acceptance criteria to support an operability determination was a
weakness. In addition, the management team was slow to understand the effects of the
July 7 biocide treatment for common mode failure. The July 22 biocide treatment,
performed to remove Zebra mussels and Asiatic clams from the SW system
inadvertently fouled the EDG 2-1 heat exchangers and made the EDG inoperable,
l
Based on information from the licensee's engineers, and their consultant, Zebra mussels
are expected to detach from the piping within 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> after the application of a toxic i
concentration of blocide. A toxic concentration was achieved in the "B" service water
header on July 7 at 1:00 p.m. (Section M2.3 discusses why a toxic concentration was not
achieved in the "A" header). However, the problem was only revealed after sufficient
flow to move the detached mussels occurred during the EDG 2-2 surveillance test.
Following cleaning and reassembly of the EDG 2-2 heat exchangers on July 16,
operators observed stable SW flow, sufficient to support EDG 2-2 operability. Based on
this information, the inspectors determined that EDG 2-2 was inoperable due to
inadequate SW flow from July 9 at 1:00 p.m. until July 16 at approximately 5:30 p.m.
7
Technical Specifications (TS) 3.8.1.1 requires both EDG's to be operable with the mactor
in Operating Mode 1. The L!miting Condition of Operation for one EDG out of service
requires the EDG to be restored to an operable condition within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Contrary te
the above, EDG 2-2 was inoperable from July 9 until July 16 due to inadequate SW fi . -
with the reactor in Operating Mode 1. The failure to meet TS 3.8.1.1 is an apparent
violation. (eel 50 412/99 07 03)
c. Conclusions
Macro biological fouling (biofouling) in the service water piping that supplies the diesel
generator was not detected during a biocide treatment on July 7. Seven days later, a
rapid and substantial degradation of service water flow occurred during an unrelated
diesel generator surveillance test. The management team was slow to understand the
effects of the July 7 biocide treatment for common mode failure and failed to protect EDG
2-1 from heat exchanger fouling. This series of events resulted in Emergency Diesel
Generator 2-2 being inoperable for longer than the 72-hour allowed outage time of the
Technical Specifications. Additionally, the initial plan for EDG 2-2 operability restoration
was incomplete, until challenged by the inspectors. This apparent violation of Technical
Specifications is being considered for escalated enforcement action.
M2.2 Proaram for Prevention of Macro Bioloaical Foulina
a. Insoection Scope (62707,93702)
l
The service water system has periodically been treated for biofouling since 1995. On '
July 7, the service water system was chemically treated. Seven days later, EDG 2-2 was
declared inoperable after a rapid service water flow reduction occurred during a
surveillance test of the EDG. Subsequent inspection of the EDG's heat exchanger
revealed an accumulation of Zebra mussels that had been dislodged from the service
water piping by a chemical treatment performed on July 7. The inspectors reviewed the
licensee's program for prevention of biofouling, the availability of industry information,
and potential precursors for this event,
b. Observations and Findinas
in 1990, the licensee recognized the potential for the plant to be affected by Zebra
mussels and pro-actively assigned an individual to obtain industry information and
develop a comprehensive strategy. Information was gathered from industrial sites in
proximity to the Beaver Valley plant and from plants that were experiencing more
advanced Zebra mussel infestations. A Zebra Mussel Working Group was formed which
included representatives from Operations, System Engineering, Chemistry and an
outside consultant. The group completed a Zebra Mussel Control Plan at the same time ;
the first Zebra mussel was identified at the intake structure in October 1995. l
The 1995 Zebra Mussel Control Plan included cleaning the service water intake bays, I
routine biocide treatments and bulk blocide treatments. The proposed " routine" blocide
treatments consisted of short duration applications (2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />), performed several times ;
4
1
.
d
8
per week, to prevent immature mussels from attaching and growing within the piping
system. The " bulk" treatments were intended to kill any attached mussels before they
grew large enough to affect service water system components. The plan also identified
the optimum river water temperatures for Zebra mussel growth. However, the Working
. Group did not establish requirements for the scheduling or frequency of the bulk
treatments. The inspectors also noted that the plan did not contain any measures to
assess or validate the effectiveness of the treatments.
A sharp increase in Zebra mussel density at the service water intake bays was identified
by the licensee in February 1998 and was documented in CR 980451. The Zebra
Mussel Working Group was convened to re-assess the 1995 recommendations.
However, the inspectors determined that the recommendations did not change based on
a review of the meeting minutes and discussion with the group's leader.
The mussels removed from the EDG 2-2 heat exchanger were up to 1.25 inches in
length. The licensee, with the help of a consultant, determined these mussels were !
approximately 1.5 years old (indicating they originated around February 1998). The j
licensee's review of the biocide treatment over the past 2 years found that the routine
biocide treatments were not consistently performed. In addition, the bulk biocide
treatments, such as the one performed on July 7, were not being performed frequently
enough to be effective. From 1995 to 1997, the Unit 2 service water system received 1
two treatments per year. In 1998 and this year, only one treatment was performed.
The inspectors determined that the Zebra Mussel Control Plan and Working Group's
meeting minutes both reflected a good awareness of industry experience. However, the
licensee did not take adequate steps to implement the corrective actions outiined in the
Zebra Mussel Control Plan in that the frequency for application of the routine and bulk
biocide treatments was not specified. In addition, the 1998 increase in Zebra mussel
population at the service water intake structure was a missed opportunity to identify the
need for more prescriptive action. The failure to implement adequate corrective actions
to prevent the intrusion and accumulation of Zebra mussels in the service water system
is an apparent violation of 10 CFR 50, Appendix B, Criterion XVI, " Corrective Actions."
(eel 50-412/99-07-04)
Senior site management involvement after the plant shutdown on July 18 resulted in a I
more focused review of the biocide program. Emphasis was placed on protecting safety i
related trains in the future by not treating both trains simultaneously. These deficiencies
and other improvements were identified and captured in CR 991848. Although the
details of the licensee's long term corrective action plans were not available at the close
of this inspection period, reasonable short term actions, which included multiple bulk ,
I
biocide treatments of the service water and EDG branch lines with flushing, cleaning and
extended flow monitoring, had been implemented to ensure service water system
operability for restart of the plant.
.
.
9
c. Conclusions
in 1995, the licensee developed a plan for the prevention of biofouling in the service
water system. Although plans for the type of biocide treatments were established,
frequencies for those treatments were not included in the plan. Subsequently, the
licensee failed to perform these treatments consistently and frequently enough to be
effective. An increase in the Zebra mussel population at the service water intake
structure in 1998 was a missed opportunity to identify this problem. This apparent
violation of Quality Assurance requirements for Corrective Action is being considered for
escalated enforcement action.
M2.3 Service Water Chemical Treatment Procedure
a. Insoection Scope (62707,93702)
On July 14, a rapid and significant service water flow degradation occurred in the EDG 2-
2 heat exchanger due to an accumulation of biofouling (primarily Zebra mussels). The
inspectors reviewed the chemical treatment procedure (2-OM-30.4.M, Revision 7) to
evaluate the adequacy of the procedure and the controls applied to this maintenance
activity.
b. Observations and Findinas
The service water system at Beaver Valley Unit 2 consists of two redundant trains. Each
train consists of a 36 inch main header that supplies large heat loads and a 12 inch
branch line that supplies smaller safety-related loads such as the EDGs, safeguards
area room coolers, and the control room ventilation. EDG 2-1 and 2-2 are supplied from
the "A" and "B" service water headers, respectively.
The service water system chemical treatment process is initiated by injecting a biocide
(Betz Deabom Powerline 3627) into the "A" and "B" service water pumps' suction. After
the biocide injection is started, its concentration is adjusted based on samples taken
from the main service water headers, downstream of the component cooling heat
exchangers. Operators then align various standby heat exchangers for flow so that they
are treated with the biocide. This process did not provide for flushing of standby heat l
exchangers (including the EDGs') following exposure to the blocide.
On July 21, after the plant was shut down, the licensee performed another bulk biocide
treatment of the service water system. On July 22, service water flow was established
through EDG 2-1 and its heat exchanger rapidly became fouled with Zebra mussels. i
The licensee did not expect this to hapoen because they believed the biofouling was an
isolated problem in the "B" service water header's branch line to EDG 2-2.
i
Subsequently, the licensee realized that on July 7, a chemistry technician failed to follow
the procedure for sampling of the service water system during a bulk chemical treatment '
activity. As a result, the intended blocide concentration was not applied to the "A" train of i
l
.
10
service water. This error was fortuitous because it precluded the simultaneous
degradation of both emergency diesel generators. However, this error was the result of
a procedure violation.
Chemistry procedure C.M. 2-3.79C, " Service Water System," Revision 1, requires
personnel to " contact the control room to determine which (CCP) heat exchanger (s) are
in service..." when sampling the service water system. On July 7, the technician did not
contact the control room and incorrectly presumed which heat exchangers were in
service. The failure to follow procedures for maintenance on safety-related equipment is
a violation of T.S. 6.8.1.a. This Severity Level IV violation is being treated as a Non-
Cited Violation, consistent with Appendix C of the NRC Enforcement Policy. This
violation is in the licensee's corrective action program as CR 991801.
(NCV 50 412/99-07-05)
The inspectors' review of the chemical treatment procedure found that it did not provide
quantitative or qualitative acceptance criteria that would ensure required flow to the
EDGs after the bulk biocide treatments. Cumulatively, the licensee's failure to properly
implement the treatment program, coupled with the procedure guidance that called for
simultaneous treatment of both service water headers, created a credible potential for
common mode failure of the EDGs. Monthly surveillance tests for the EDGs are
performed on a staggered basis (one EDG is run every 14 days). Because the bulk
treatments were not scheduled in relationship to the EDG surveillances,14 days could
have passed before a routine EDG surveillance would have detected a common mode
problem.
The inspectors determined that a common mode service water system failure, coupled
with a 14-day period prior to discovery, would result in a significant reduction of safety
margins. It was fortuitous that EDG 2-2 was tested only 7 days after the bulk biocide
treatment; and, it was also fortuitous that both service water trains were not chemically
treated, as planned.
10 CFR 50 Appendix B, Criterion V, " Instructions, Procedures and Drawings," requires in
part, that, procedures include appropriate acceptance criteria for determining that
important activities have been satisfactorily accomplished. Chemical treatment
procedure 2-OM-30.4.M, Revision 7, did not contain quantitative or qualitative
acceptance criteria for service water flow to the EDGs nhor the treatment activity. The
failure to establish adequate acceptance criteria for tNa procedure is an apparent
violation of 10 CFR 50 Appendix B, Criterion V. (eel 50-412/99-07-06)
c. Conclusions
The licensee failed to provide adequate acceptance criteria in its procedure for bulk
chem! cal treatments of the senrice water system. Specifically, the emergency diesel
_ generators were not monitored to assess the impact of biofouling dislodged during the
treatment. The lack of acceptance criteria, coupled with the simultaneous treatment of
both service water trains, created the potential for a common mode failure and a
significant reduction in safety margins. The failure to develop an adequate procedure for
11
bulk chemical treatments is an apparent violation of Quality Assurance requirements and
is being considered for escalated enforcement action.
On July 7, a chemistry technician failed to follow the procedure for sampling of the
service water system during a bulk chemical treatment activity. As a result, the intended
biocide concentration was not applied to the "A" train of service water. This failure to
follow procedures is being treated as a Non-Cited Violation, consistent with Appendix C
of the NRC Enforcement Policy.
Ill. Enaineerina
E1 Conduct of Engineering
E1.1 Root Cause Determination for EDG 2-2 Failure
a. Inspection Scoce (93702.)
The inspectors reviewed engineering documents and field practices to assess the
licensee's root cause determination and corrective actions in response to the failure of
EDG 2-2 and tripping of the 4 kV tie breaker 2F7. In evaluating the licLnsee's response
to this event, the inspectors attended several licensee management meetings, system
engineering troubleshooting meetings, and reviewed the ERT report, and CR 991749
entitled " Loss of Unit 2,2DF Emergency Bus."
b. Observations and Findinos
Overall, the inspectors determined that the licensee's root cause analysis was conducted
in a professional and timely manner. The licensee's troubleshooting efforts were
hampered by the fact that they were unable to reproduce the failure during static testing
or subsequent EDG runs (loaded and unloaded). The voltage regulator and associated
~
sub-components performed properly during a half hour run on July 17,1999, and an ,
extended fullload run on July 21,1999. The licensee promptly sought support from
several vendors to assist in the diagnosis of the EDG failure. The licensee's
detemiination that the voltage regulator lost communication from the Signal Mixer Output
Card was independently supported by the vendors. The inspectors observed that the
licensee was very effective in the management of the vendor support.
The licensee concluded that the initiating eve. A was a loss of the voltage regulation due
to an intermittent control relay contact in the EDG 2-2 voltage regulator power amplifier.
The loss M the volta 0e regulation caused the generator output power factor to increase.
The increasing power factor resulted in a rise in ground current above the expected ,
setpoint value of protection relay 50-VF207G. This relay in turn opened the 2F7 tie j
, breaker, separating the emergency bus (2DF) and EDG from the nonsafety-related bus l
(2D) and %6 gde
l
l
.
12
Through extensive engineering and troubleshooting the licensee determined that an
intermittent control relay contact in one of three control relays,90FC, 90X or 43(K1),
could have resulted in the observed event. The licensee replaced the three suspected
relays and retested EDG 2-2 (Work Order 99-213186-000). The inspectors reviewed the
retest requirements and observed portions of the retest and found the licensee's actions
to be acceptable.
The licensee recommended long term corrective actions under CR-991749. These
recommended actions include changes to the EDG monthly surveillance procedures that
will provide performance trending and monitoring for the suspected relays. Additionally,
the licensee has planned to perform a failure analysis for the removed relays and review
the results with the vendor to determine if additional preventive maintenance should be
incorporated.
c. Conclusion
The licensee identified the most probable cause of the EDG 2-2 failure on July 16 as an
intermittent control relay contact in the voltage regulator circuit. A questioning attitude
throughout the engineering evaluation and root cause analysis was observed, and the
licensee made effective use of the vendor support. The 'voltage regulator repairs and
retest were appropriate. Long term corrective actions recommended by the Event
Response Team are reasonable actions to prevent recurrence of the relay failure.
E1.2 Effects of Under Voltaae on Emeroency Bus 2DF
a. Inspection Scoce (93702)
The inspectors reviewed the actual response of electrical equipment powered from the
2DF bus during the EDG 2-2 failure and compared it to the expected response. The
inspectors also reviewed the licensee's investigation and final disposition regarding the
condition of equipment affected by this event.
b. Observations and Findinas
i
l
A low voltage condition existed on the 4 kV emergency bus (2DF) from the time the 4 kV
'
tie breaker automatically tripped until operators opened the EDG 2-2 output breaker.
The voltage on bus 2DF stabilized at approximately 2100 volts (% of rated voltage) and !
lasted for a duration of approximately 75 seconds. The expected response of a motor to
a decrease in voltage would be to lose speed, stall or overload. At the start of the event,
Service Water Pump "B" and the Charging Pump "B" were running. During the low l
voltage condition the two pumps slowed down, as was expected. This was indicated by '
their respective control room alarms, "SWS HDR B PRESS LOW' and "RCP 21 A SEAL
INJ FLW LOW" l
During the transient, Safety injection Pump "B" and Component Cooling Pump "B"
received expected start signals and sequenced onto the bus due to the degraded
voltage condition. The licensee determined that the pumps' breakers closed and the
I
.
13
pumps probably started, but did not reach full speed due to the low voltage condition.
During starting, the motors behave as constant ir ;sdance loads. This reduces the
starting current proportional to the motor terminal .< Mage. The fact that the motors did
not trip supports the licensee's evaluation that the motors probably started but did not
reach full a, peed. Extended operation of these motors at reduced voltage possibly would
have resulted in the motors tripping on over-current. Due to the short time that these
motors were operated in a low voltage condition, no over-current trips were observed.
In addition to the engineering evaluations, the licensee satisfactorily completed electrical
testing (bridge and megger checks) of the affected equipment under work orders 99
213211-000,9921259-000,99-123266-000 and 99-213267-000. The inspectors found
that the licensee's conclusions were technically sound and that they promptly confirmed
the condition of the affected equipment.
c. . Conclusion
Equipment aligned to the 4 kV emergency bus that was affected by the EDG 2-2 failure
responded as expected during the low voltage condition which existed for approximately
75 seconds. The licensee's engineering evaluatioi.: for loads that were running, or
received start signals, were technically sound. In addite electrical tests were used to
confirm the condition of the affected equipment. l
E1.3 Unexpected Restad of EDG 2-2
a. Inspection Scope (93702)
On July 16, EDG 2-2 unexpectedly started when its control circuit was placed in the local
control mode during recovery from the electrical transient. The inspectors walked down
portions of the EDG 2-2 starting system, reviewed engineering documentation, and
observed portions of the field investigation to assess the licensee's root cause
determination and corrective actions.
b. Observations and Findinas
On July 16, during the emergency bus 2DF degraded voltage event, EDG 2-2 was
manually secured from the control room by means of the emergency stop buttons.
Operators decided that the engine should be secured locally to minimize the possibility of
spurious restarts. An equipment operator was directed to place the " Auto - Local" switch
at the diesel control panel in the " Local" position. This is a two position switch that, by l
design, will block all automatic EDG start signals in the " Local" position. However, the l
EDG immediately restarted when the control switch was placed in the " Local" position. !
Minutes later, the local control switch was retumed to the " Auto" position and the EDG !
was secured from the control room.
The licensee promptly initiated an investigation of the event as described in the Event l
Response Team charter. The licensee conducted testing of the switch circuit (CR
991755 and wo* order 99-213246-001). The testing included wiring verifications and
i
.
14
)
switch functional testing. Additionally, the licensee verified the functionality of the start
circuit by placing the local control switch in the " Local" position and inserting an
emergency start signal. The circuitry functioned as designed and did not start the
engine. The licensee determined that no deficiencies existed in the starting circuit or
local control switch.
The licensee concluded that a momentary breaking and remaking of the " Auto" contacts
occurred in the local control switch while it was being placed in the " Local" position. This
was identified as the most likely cause of the unexpected EDG start. Switch operating
technique may have caused the contact cycling. However, this could not be confirmed
based on interviews with the operator. The licensee initiated long term corrective actions
under CR 991748 to review this event, and proper switch operation technique, in
operator training.
The inspectors interviewed the system engineer and reviewed the test results against the
Updated Final Safety Analysis Report (UFSAR) logic diagram numbers 7.4-31 through
7.4-37 and EDG 2-2 elementary diagram numbers 10080-E-12K,12241-E-12M8 and
12241-E-12N9. The inspectors found the test results to be consistent with the diagrams,
c. Conclusion
The licensee conducted adequate troubleshooting of the EDG 2-2 local control switch
anomaly that resulted in an unexpected start. Intermittent contact resistance during
operation of a local control switch was identified as the most probable cause, after other
possibilities were eliminated through testing and design reviews. Appropriate short term
actions were completed; and, long term corrective actions will be addressed by the
licensee's corrective action process.
E1.4 Loss of Charoino for Batterv 2-2 and Batterv 2-4
a. Insoection Scooe (93702) .
l
The inspectors interviewed the system engineer and reviewed the event operations logs I
and surveillance test to determine if the battery performed as expected and if the
licensee properly executed the Technical Specification (TS) requirements for the DC
system,
b. Observations and Findinos
l
The loss of the emergency bus 2DF at 5:29 p.m. on July 16 resulted in the loss of power
to the chargers for two safety-related batteries. Batteries 2-2 and 2-4 supply vital buses
generally associated with the "B" train of safety-related plant equipment.
The licensee promptly recognized that this condition required a plant shutdown. A loss
of two battery chargers is outside the scope of TS requirements for the DC system, and
therefore TS 3.0.3 required operators to place the reactor in the " hot standby" condition
within 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br />. Without the chargers, the batteries began discharging to supply their
15
respective DC loads. After approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />, power was restored to the chargers.
However, neither battery could meet the operability limits of TS 3.8.1 and therefore the
plant shutdown continued. Battery 2-4 recovered to above the TS Category "B" limits for
operability after approximately 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and 8 minutes, and Battery 2-2 after about 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />
and 43 minutes. The shutdown due to battery related problems was terminated at 10:18
p.m. on July _16.
The surveillance data available for this event was collected after the battery chargers
were returned to service and therefore loaded battery data was not captured. Post ,
discharge surveillances showed that Battery 2-2 pilot cell (#49) read 2.0055 VDC with a !
specific gravity of 1.192, and Battery 2-4 pilot cell (#9) read 2.0135 VDC with a specific
gravity of 1.208. The licensee estimated that the load on the batteries during the three
hours of the event was approximately 150 amps for the 2-2 battery and approximately 40
amps for the 2-4 battery. These discharge rates are within the battery design, as
specified in the 125 volt DC Design Document (2-DBD-39, Revision 3, Section 5.1.1.b)
and the Updated Final Safety Analysis Report (Section 8.3.2.1.3).
The design basis for BV Unit 2 includes the ability to withstand the single failure of one
battery or one DC bus. During the July 16 event, two battery chargers became
inoperable and their associated batteries began to discharge. A TS 3.0.3 shutdown was
initiated in response to this event. At some point in time, the two batteries became
" inoperable" when their pilot cell voltage and specific gravity decreased below TS
requirements. However, during the July 16 event both batteries remained functional until
their chargers were restored. All of the instruments or relays powered from the batteries
remained in their apprcpriate state.
Battery 2-2 and 2-4 are designed for a 2-hour discharge under design basis accident
conditions. If the battery chargers are not recovered, it may be possible for the two
batteries to discharge below the point of being functional during the 7 hours8.101852e-5 days <br />0.00194 hours <br />1.157407e-5 weeks <br />2.6635e-6 months <br /> that TS 3.0.3 I
would allow to reach hot standby, This issue was discussed with licensing and
engineering personnel during a conference call on August 5. The potential for two
batteries to become non-functional prior to completing the plant shutdown is being l
evaluated by the licensee. An Inspector Follow-up item will be opened to track NRC
review of the licensee's final evaluation. (lFl 50-412/99-07-07)
c. Conclusion
The safety-related 125 volt batteries and DC system responded normally during the loss
of power to their chargers. TS requirements for the DC distribution system were
appropriately implemented. The licensee's engineers provided a reasonable
assessment of the batteries' performance during this ewnt and concluded that the
battery discharge rates were within their design. An inspector follow-up item was
opened to review the DC system's capability during a loss of battery charging event that
requires a plant shutdown.
!
,
,
16
E1.5 Loss of Seal Coolina for Two Reactor Coolant Pumos
a. Inspection Scooe (93702)
The inspectors reviewed the licensee's engineering evaluation for continued operation of
'ne reactor coolant pumps after the coincident loss of seal injection and thermal barrier
flow. The inspectors also conducted an independent assessment of the risk associated
- v&h the concurrent loss of both cooling sources.
b. phy rvations and Findinas
After restoring charging flow using the "A" charging pump, operators discovered that
both seal injection and thermal barrier cooling had been lost for almost three minutes to
the "B" and "C" RCPs. An engineering evaluation was performed based on
temperatures and flows recorded before and after the event.
Pre-Event Readina (a 5:00 om)
Seal #1 leakoff temperature: 150 165 167 <210*F
Pump bearing temperature: 126 139 131 <184*F
Seal #1 leakoff flow: 2.4 2.3 2.7 1 - 5 gpm
Event and Post-Event Maximum Temperatures
and Minimum Flow (5:29 om- 9:00 nm)
Seal #1 leakoff temperature: 153 175 175 <210*F ,
Pump bearing temperature: 130 148 139 <184*F l
Seal #1 leakoff flow; 2.2 1.7 2.5 1 - 5 gpm l
Post-Event Steadv State Readinas (10:00 om)
Seal #1 leakoff temperature: 147 167 167 <210*F
Pump bearing temperature: 123 138 132 <184*F
Seal #1 leakoff flow: 2.5 2.3 3.0 1 - 5 gpm
Pre and post-event readings were consistent, and the transient readings were well within
the allowable limits. In addition, the licensee ensured that the limits being used for the
allowed time without seal injection and thermal barrier flow were consistent with the
guidance provided by the vendor (Westinghouse) and an independent contractor
(Brookhaven National Laboratory). The licensee's evaluation concluded that the RCP
seals and the pump bearings suffered no consequences from the short duration loss of
cooling. The inspectors assessed the engineering evaluation, independently reviewed
the pump parameters and the vendor and contractor recommendations, and considered
the conclusion to be acceptable.
17
The inspectors noted the BV Unit 2 RCP seals have high-temperature O-rings and
floating ring sea!s. This information has not been credited in the Unit 2 Probability Risk
Assessment. The extent to which these components may have played a roll in the
minimal change in seal temperatures and seal leakoff had not been evaluated by the
licensee at the conclusion of the inspection. The licensee's efforts for the short term
were focused on the current seal condition. During the Nuclear Safety Review Board
review of the ERT report, the licensee's design engineers indicated that a re-evaluation
of the CDF contribution for RCP seal LOCAs would be considered. The licensee's
preliminary evaluation indicated that the RCP seal LOCA contribution to CDF should be
reduced. The licensee's final evaluation of the RCP seal performance during this event
will be reviewed by the NRC at a later date. (IFl 50-412/99-07-08)
The ERT identified that the component cooling supply valves for the RCP's thermal
barriers are DC powered at BV Unit 1. This design difference woul'd preclude the total
loss of RCP seal cooling should a loss of one 4 kV emergency bus occur on Unit 1.
Condition Report (CR) #991818 was initiated to review the potential design deficiency at
Unit 2. As an immediate corrective action, the associated surveillance test procedures
were revised to ensure the in-service charging pump is powered from the opposite train
during EDG testing.
c. Conclusion
The licensee appropriately evaluated the operational condition of the reactor coolant
pump seals after the loss of all seal cooling (concurrent loss of both seat injection and
thermal barrier flow). Based on plant data taken before, during, and after the event, the
licensee determined no significant heat up of the seals occurred.
V. Manaaement Meetinas
X1 Exit Meeting Summary
The team met with licensee representatives periodically throughout the inspection and at a
debriefing prior to leaving the site. Following the in-office review of materials collected during
the inspection, and exit meeting was conducted on July 29,1999 by telephone call. At that time,
the purpose and scope of the inspection were reviewed, and the preliminary findings were
presented. The licensee acknowledged the preliminary inspection findings. ;
--
.
18
ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
eel 50-412/99-07-03 Diesel Generator Inoperable for Longer Than 72-hours
eel 50-412/99-07-04 Failure to Implement Corrective Actions for Biofouling
eel 50-412/99-07-06 Inadequate Procedure for Chemical Treatment
IFl 50-412/99-07-07 Battery Capability During Shutdown for Two Inoperable Chargers
IFl 50-412/99-07-08 RCP Seal Performance During July 16,1999 Loss of Seal
Cooling
Ooened/ Closed
NCV 50-412/99-07-01 Failure to implement Alarm Response Procedure
NCV 50-412/99-07-02 Failure to Develop Procedure for Loss of Emergency Power
NCV 50-412/99-07-05 Failure to Follow Procedura for Service Water Sampling
1
i
-
c-
.
.
t 19
LIST OF ACRONYMS USED
AC Altemating Current
ANSS Assistant Nuclear Shift Supervisor
AOP Abnormal Operating Procedure
ARP Alarm Response Procedure
'BV Beaver Valley
CDF Core Damage Frequency
CFR Code of Federal Regulations
CR Condition Report
DBD Design Basis Document
DC Direct Current
DLCo Duquesne Light Company
EDG Emergency Diesel Generator
eel Escalated Enforcement item
ERT Event Response Team
IFl inspector fo!iow-up item I
kV kilovolt !
LCO Limiting Condition for Operation I
LOCA Loss of Coolant Accident
NCV Non-cited Violation
NNS Nuclear Shift Supervisor
NRC Nuclear Regulatory Commission
PO Plant Operator
PRA Probability Risk Assessment
RCP Reactor Coolant Pump
RO Reactor Operator
TS Technical Specification
UFSAR Updated Final Safety Analysis Report
URI unresolved item
VIO ' Violation
I
n
_
.
ATTACHMENT A '
July 23,1999
MEMORANDUM TO: Wayne D. Lanning, Director, Division of Reactor Safety
A. Randolph Blough, Director, Division of Reactor Projects
FROM: Hubert J. Miller ORIGINAL SIGNED BY:
Regional Administrator
SUBJECT: SPECIAL INSPECTION TEAM CHARTER - FAILURE TO TRIP
REACTOR AS REQUIRED BY ALARM RESPONSE
PROCEDURE AND MULTIPLE EQUIPMENT PROBLEMS AT
BEAVER VALLEY UNIT 2
You are directed to perform a Special Team Inspection review of the causes, safety implications,
and associated licensee actions which led to multiple equipment problems and the failure of
operators to trip the reactor as required by an alarm response procedure at Beaver Valley Unit 2
on July 16,1999. The basee for the NRC concern involve several equipment and operator
performance issues that are not currently understood.
DRS is assigned responsibility for the overall conduct of this inspection. DRP is assigned
responsibility for resident inspector and clerical support and coordination with other NRC offices.
Brian McDermott is designated as the onsite Team Leader. Team composition is described at
the end of this memorandum. Team members will work for Mr. McDermott and are assigned to
this task until the report is completed.
OBJECTIVES
The general objectives of this team inspection are to:
a. Conduct a timely, thorough, and systematic review of the circumstances surrounding the
events, including the sequence of events that led to and followed the July 16,1999, loss
of the 2DF emergency bus which resulted in the interruption of RCP seal water flow and
thermal barrier cooling.
b. Collect, analyze, and document relevant data and factual information to determine the
conditions and circumstances surrounding these equipment and operator performance
issues, and the appropriateness of the response to the events by the licensee's
operating staff.
c. Investigate the equipment issues which occurred prior to, during, and after the loss of the
emergency bus; including failures or problems with the EDG, emergency bus and
, breakers, service water flow, battery and chargers, and reactor coolant pump seal
cooling flow.
_
I-
- -
.
Wayne D. Lanning - 2
A. Randolph Blough l
d. Assess the safety and risk significance of the events and communicate to Regional and !
Headquarters management the facts and any safety concems related to the problems
identified.
e. Evaluate licensee management's review of and response to the events and corrective
actions taken.
SCOPE OF THE INSPECTION
The team should identify and document the relevant facts and determine the probable causes of
the events. It should also critically examine the licensee's response to the event.
The team should:
a. Develop a detailed chronology of the events.
b. Identify safety issues, problems and concerns with equipment, personnel performance,
processes and procedures, and management response.
Potentialitems to be considered:
e Licensee staff actions before, during, and following the events, including operator
actions, procedure adequacy and usage during a loss of RCP seal cooling.
e' Configuration controls, including previous modifications, and event related
system alignments.
l
I
e Management oversight and administrative controls in place be'iore, during, and
following the events.
l e Coordination of maintenance and operations activities before and during the l
l events.
l e Adequacy and implementation of troubleshooting procedures.
c. Determine the root causes of the events, where practicable, as a result of the team's
l evaluation and document equipment problems, faiiures, and/or personnel errors which
i directly or indirectly contributed to the event or complicated the response.
d. Determine the expected response of the plant due to the loss of the vital bus and
compare it to the actual response.
- e. Determine the adequacy of the initial responses of the operations and technical support
staffs to the events.
l
l
,
.
3
f. Determine the effectiveness of the Event Review Team (ERT) assessment and
evaluation of the events and the subsequent technical specification required shutdown,
including the ERT effectiveness in communicating the results to licensee management.
g. Understand the scope and direction of the licensee's ERT. Assess the quality of the
investigation, including disposition of issues and findings of the ERT. Determine
management's use of ERT findings relative to restart of Unit 2, continuing operation of
Unit 1, and identification of effective corrective actions.
h. Determine the relationship of previous events or precursors, if any, to this event,
including actions taken to address service water heat exchanger fouling by clam and
musse shells,
i. Verify that the licensee has identified and effectively addressed issues that affect start-up
of Unit 2 and the continuous operation of Unit 1.
J. Determine the potential generic implications of this event.
CONDUCT OF THE INSPECTION
The team should understand the scope and direction of the licensee's investigation and
assessments of the events, and their initial response. Through sampling and independent
verification, the team should use information collected by the ERT. The pace and nature of
team activities should be gauged to assure its assessments are independent of the licensee's
and that, where practicable, they do not unduly impact licensee efforts.
SCHEDULE
The Special Inspection Team shall be dispatched to Beaver Valley, Unit 2 so as to arrive and
commence the inspection on July 20,1999. A written report on this inspection shall be provided .
to me within 45 days of completion of the onsite inspection. The team leader shall provide daily l
status calls to the team manager.
TEAM COMPOSITION
The assigned team members are as follows:
Team Manager: Wayne Lanning, DRS
Onsite Team Leader: B. McDermott, DRP
Onsite Team Members: B. Norris, DRS
C. Cahill, DRS
G. Wertz, DRP
..
ATTACHMENT B
4
Beaver Valley EDG 2-2 Cooling Degradation and Electrical Failure
Date Time Event
_
7/7 Unit 2 service water (SW) trains "A" and "B" treated for biofouling
7/14 EDG 2-1 Surveillance Test (~ 2 hr. run), supplied by "A" SW train.
2132~ EDG 2-2 Surveillance Test begins, supplied >1500 gpm by "B" SW train.
2147 . Service water flow to EDG 2-2 decreases from initial reading to 1070 gpm
(1170 gpm design basis minimum flow)
2157 EDG 2-2 secured and declared inoperable, Technical Specification (TS) 72 hr
period to restore the EDG begins
7/15 1141 Three gallons of zebra mussels removed from the EDG 2-2 heat exchanger
7/16 1324 SW flow through EDG 2-1 initiated for confidence check based on zebra
mussels discovered in EDG 2-2
!
1628 EDG 2-2 started and loaded to 4400 KW for post maintenance test following
heat exchanger cleaning
.
1729 Normal power supply breaker to emergency bus 2DF trips open
l
unexpectedly. Bus 2DF and EDG 2-2 voltage observed at half of normal.
1730 Operator opens EDG 2-2 output breaker based on degraded voltage and
l manually trips EDG from control room
1730 Emergency bus 2DF de-energizes, major equipment affected: running
j- charging pump "B" lost resulting in loss of all reactor coolant pump (RCP)
'
seal injection flow; component cooling to "B" and "C" RCPs isolates; and
battery chargers for batteries 2-1 and 2-1 de-energized.
1730 Vital DC buses 2-2 and 2-4 auto swap to battery supply with no loss of
function.
TS 3.0.3 is entered due to loss of two battery chargers.
~1733 "A" charging pump manually started, seal injection flow restored to all RCPs.
1800 Emergency bus DF loads put in pull-to-lock in preparation for restoration.
1815 EDG 2-2 put in " Local" by operator at local panel and EDG auto starts.
Operator is unable to trip EDG from local panel. " Local-Auto" switch put back
in " auto" and EDG is secured from the control room
j 1912 Load reduction commenced for TS 3.0.3 required shutdown
'
1941 Normal power restored to emergency bus 2DF and loads are restored during
the next 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br />.
2030 Charger for Battery 2-4 restored
.
I
.
2
- 2035 Charger for Battery 2-2 restored
2115 Batteries 2-2 and 2-4 tested. Did not meet TS Category "B" values required
for operability. TS 3.0.3 shutdown continued. j
2138 Battery 2-4 meets Category "B" allowable values.
TS 3.0.3 exited, TS shutdown still required for 1 battery inoperable.
2218 Battery 2-2 meets Category "B" allowable values.
Plant shutdown terminated, reactor power held at - 60%.
TS 72 hr clock for EDG 2-2 restoration still counting down.
7/17 2021 Plant shutdown continued, based on expectation that EDG 2 2 would not be
l restored within the TS allowable 72 hr period.
2157 TS 72 hr period to restore EDG 2-2 ends,6 hr clock to be in Hot Standby 4
begins.
7/18 0206 Plant is in Mode 3, Hot Standby
1721 Plant is in Mode 5, Cold Shutdown
'
l
l
i
l
l
1
l
u