IR 05000361/1999003
| ML20206B484 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 04/20/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML20206B427 | List: |
| References | |
| 50-361-99-03, 50-361-99-3, 50-362-99-03, 50-362-99-3, NUDOCS 9904290259 | |
| Download: ML20206B484 (25) | |
Text
_.
'
.
.
ENCLOSURE U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
,
Docket Nos.: 50-361 50-362 License Nos.: NPF-10 NPF-15 Report No.: 50-361/99-03 50-362/99-03 Licensee: Southern California Edison C Facilit San Onofre Nuclear Generating Station, Units 2 and 3 Location: 5000 S. Pacific Coast Hw San Clemente, California Dates: February 5 through March 12,1999 1
Inspectors: J. A. Sloan, Senior Resident inspector )
J. P. Rodriguez, Reactor Engineer Approved By: Linda J. Smith, Chief, Project Branch E, Division of Reactor Projects ATTACHMENTS:
Attachment 1: Sequence of Events Attachment 2: Figures Attachment 3: Supplementalinformation 9904290259 990420 I PDR ADOCK 0500036 G PDR
.
.
EXECUTIVE SUMMARY San Onofre Nuclear Generating Station, Units 2 and 3 NRC Inspection Report No. 50-361/99-03; 50-362/99-03 j This special inspection in'cluded aspects of licensee operations, maintenance, engineering, and plant support associated with the loss of shutdown cooling (SDC) event that occurred in Unit 2 on February 1,199 {
Operations
- Personnel and system response to the February 1,1999, loss of SDC event in Unit 2 were excellent:
=
Operators correctly and promptly classified a loss of SDC event, and their response to the event was good. Operators' execution of the " Loss of Shutdown ,
Cooling" abnormal operating instruction was excellent (Section 01.1). J
=
The local containment evacuation, although directed by procedure, was a precautionary and conservative measure. Nevertheless, it was accomplished i expeditiously and without hesitation (Section O1.1). '
=
All equipment and systems performed as designed in response to the event. A thorough inspection and testing plan for potentially affected components, including the transformers and breakers, was developed and completed, revealing no equipment damage as a result of the event. The overcurrent protection scheme for the safety related feeder breakers and Train A Bus 2A04 was acceptable, and the undervoltage protection circuits functioned as designed (Sections O2.1 and O2.2).
=
The licensee's shutdown nuclear safety program and its implementation ensured that safety functions were adequately maintained during a loss of SDC event in Unit 2 on February 1,1999. The defense-in-depth strategy in place for core and spent fuel pool decay heat removal provided reasonable assurance that boiling would not occur before a backup active decay heat removal system could be placed in service. The minimal heatup that occurred during the event demonstrated that the margin provided by the large heat sink adequately maintained the decay heat removal safety function for both the core and spent fuel pool (Section O2.1).
The shutdown nuclear safety procedure was weak in that it did not fully reflect the interrelationship between the electrical distribution safety function and other safety functions that were dependent on the electrical distribution system. Specifically, the availability of Train B Bus 2A06 was credited for fulfilling the electrical distribution safety function as Method B, but did not contribute to supporting the decay heat removal safety functions because the Train B systems were mechanically unavailable. Additional j measures to protect the Train A electrical distribution system would have been j appropriate but were not required by the procedure. This weakness was not identified j by the licensee's investigation and was not included in the corrective action program (Section O2.1).
i l
i l
J
.
.
l-2- !
Operators had not thought through, in advance, the actions that would be necessary to ;
restore SDC in the event of needing to restart the Train A saltwater cooling pum i However, the necessary information was available in procedures, and operators worked through the condition expeditiously when it occurred, resulting in only an approximate 10-minute delay in SDC cooling restoration. There was negligible safety consequence to this delay because of the large water inventory. Advance planning for those actions was not required for the circumstances of this event (Section 01.1).
One example of a noncited violation (NRC Enforcement Policy, Appendix C) of 10 CFR Part 50, Appendix B, Criterion V, " Instructions, Procedures, and Drawings," was identified because the procedural controls for ensuring that the reserve auxiliary transformer could not affect operable safety equipment were inadequate and inappropriate for the circumstances. The switching order closed the grounding disconnects before the safety-related low side feeder breakers were racked out. This established an unnecessarily risky configuration that was a substantial contributor to the February 1,1999, loss of SDC event and the subsequent inability of the emergency diesel generator to perform its design function when it was called upon. This example is ;
in the licensee's corrective action program as Action Request 990200037 (Section O3.1).
One example of a noncited violation (NRC Enforcement Policy, Appendix C) of 10 CFR Part 50, Appendix B, Criterion V, " Instructions, Procedures, and Drawings," was identified because the instructions provided to the operators and electricians for freeing I Breaker 2A0418, including discharging the closing springs, were inadequate and inappropriate for the circumstances. This inadequacy substantially contributed to i causing a loss of SDC in Unit 2 on February 1,1999. This example is in the licensee's corrective action program as Action Request 990200037 (Section O3.1):
- None of the licensee personnel involved, including outage management I personnel and various supervisors and managers, recognized that attempting to rack out the breaker, once it was known to have interference, substantially increased the risk of the activity and warranted more careful planning and control, consistent with the licensee's recently-promulgated " error-likely situations" guidance (Section O3.1),
- During prejob briefings discussing the plan to discharge the closing springs to prevent inadvertent closure, undue reliance was placed on the knowledge of one j
'
individual. The lack of specific knowledge regarding breaker operational details by others involved, especially the cognizant engineer, was not established or considered. While a procedural precaution was discussed, personnel did not confirm that the precaution did not apply, again trusting in one individual's knowledge (Section O3.1).
- A blanket maintenance order was used that provided no specific direction regarding breaker operations, and the operating instruction for 4160 volt air circuit breakers did not directly address the breaker inte'iocks (Section O3.1).
c
..
.
3-Enoineerina
- The licensee's assessment and corrective program implementation, in response to the loss of SDC in Unit 2 on February 1,1999, was good:
The root cause assessment of the stuck breaker was good. The incorrect configuration affected only the ability to rack the breaker out, which was not a safety function. The licensee's conclusion that the configuration error was the result of a vendor performance deficiency was rigorously supported, and the licensee event report correctly identified the applicability of 10 CFR Part 21. The licensee's corrective actions, including enhancing the appropriate circuit breaker inspection ptocedure, were appropriate (Section O7.1).
- The licensee's corrective action program appropriately focused on causal factors, which were clearly discussed in the Level 1 event report. Corrective actions, including several programmatic enhancements, were clearly identified and were entered into the corrective action tracking system. However, the event report investigation and report did not identify or address the safety function interrelationship weakness discussed above (Section O7.1).
- . The licensee's assessment of the safety significance of the event considered all relevant factors. The conclusion that the core damage risk increase was "very small" was correct (Section O7.1).
,
i I
t I
h
.
.
Report Details Summary of Plant Status On February 1,1999, Unit 2 was in Mode 6 during a refueling outage, and Unit 3 was operating at essentially 100 percent reactor powe l. QJLerations 01 ' Conduct of Operations 01.1 Loss of SDC - Unit 2 '
, Inspection Scope (93702) J l
After Unit 2 experienced a loss of SDC on February 1,1999, the inspectors responded to the control room, observed operator response, and reviewed operating logs, records of instrument readings of plant parameters, and the licensee procedures addressing loss of SDC, and emergency event classificatio Observations and Findinas Event Summary On February 1,1999, Unit 2 was in a refueling outage in Mode 6 with Train A SDC in ;
service and the refueling cavity flooded to 23 feet of water above the reactor vessel l flange. Spent fuel pool (SFP) cooling was being provided by the Train A containment i spray pump. Scheduled maintenance was in progress on Train B safety-related _
equipment, rendering this alternate train of SDC and SFP cooling unavailabl {
Train A 4160 volt Bus 2A04 was being powered from Unit Auxiliary Transformer 2XU '
The Unit 2 reserve auxiliary transformer (RAT),2XR1, was being cleared to support routine maintenance. The high side of the RAT was disconnected from the transmission system and connected to ground in accordance -with the clearanc During performance of the remaining clearance activities, licensee personnel were unable to rack out Breaker 2A0418, the feeder breaker to safety Bus 2A04 from the RAT, because of interference between the breaker and its cubicle. The licensee developed a plan to remove the breaker. Operators were concerned that the breaker could inadvertently close, so the plan called for verifying that DC control power was off, discharging the closing springs, inserting a tool to clear the interference, and then racking out the breaker. Licensee personnel incorrectly believed that they had identified a method for discharging the closing springs that would not close the breake At 9:59 a.m., licensee personnel discharged the closing springs, causing Breaker 2A0418 to close. The closed breaker established an indirect path to ground through the RAT, resulting in a high load on Bus 2A04 that rapidly drew the bus voltage down. The fault was cleared by the loss of voltage relays on Bus 2A04. All of the feeder breakers to Bus 2A04 tripped open except Breaker 2A0418, as designed. This breaker remained in the off-normal configuration (closed) because the control power
.
.
-2-fuses had been removed as described above. As designed, the emergency diesel generator (EDG) for Bus 2A04 started but did not tie to the bus because of a protective interlock that prevented more that one feed to the bus at a time. These conditions resulted in a loss of Train A equipment, including SDC and SFP coolin Operator Response Operators responded to the event by promptly implementing Abnormal Operating Instruction SO23-13-15, Revision 10 " Loss of Shutdown Cooling,"immediately directing that core alterations be suspended. Communications among operators were clear, and all objectives of the procedure were accomplished in a methodical and controlled manne Train A EDG 2G002 started, but the output breaker did not close, because of a protective interlock that prevented more than one feed to safety Bus 2A04. Operators promptly recognized this condition, determined it to be appropriate for the circumstances, and secured the diese At approximately 10:06 a.m., operators directed Health Physics to conduct a local evacuation of containment, as directed by Abnormal Operating instruction SO23-13-1 Approximately 50 personnel were in containment at the time of the event. Operations and Health Physics personnel recognized that there was no immediate risk to personnel, as the time to boil was approximately 28 hours3.240741e-4 days <br />0.00778 hours <br />4.62963e-5 weeks <br />1.0654e-5 months <br />. However, personnel readily complied with the order to evacuate. Containment evacuation was completed at approximately 10:18 a.m., and containment closure was achieved at 10:28 Operators declared an Unusual Event at 10:09 a.m., based on the loss of SDC for more than 10 minutes. This classification was consistent with Emergency Plan implementing Procedure Vill-1, Revision 9," Recognition and Classification of Emergencies," and the declaration was timel Operators and electricians worked to restore power to Bus 2A04. No ground alarms or overcurrent indications were identified so, after manually opening Breaker 2A0418, operators reenergized Bus 2A04 by closing the feeder breaker from the unit auxiliary transforme In order to restnre SDC, operators had to first restore saltwater cooling to Train Train A had been operating with the heat exchanger discharge flowpath aligned to overboard to the beach,instead of the normal alignment of discharging to the Unit 2 outfall, because the Unit 2 intake structure was dewatered in support of outage maintenance activities. Because of this alignment, the pump did not start when operators attempted to start it from the switchgear room. Operators researched Operating Instruction SO23-2-8, " Saltwater Cooling System Operation," Revision 19, and determined that an interlock prevented starting the pump with the abnormal alignment, but that actuating the fire isolation switch for the pump would bypass the interlock. Operators had to get the key to the fire isolation switch cabinet and actuate the switch before the pump started. These actions resulted in a short delay in restoring SD m
.
L -3-Operators had not thought through, in advance, the actions that would be necessary to restore SDC in the event of needing to restart the Train A saltwater cooling pum However, the necessary information was available in procedures, and operators worked through the condition expeditiously when it occurred, resulting in only an approximate 10-minute delay in SDC restoration; SDC was restored at 10:25 a.m., approximately 12 minutos after power was restored to Bus 2A04. There was negligible safety
' consequence to this delay because of the large water inventory. Advance planning for these actions was not required for the circumstances of this even Operators mon;tored core and SFP temperatures and determined that these temperatures increased approximately 5'F and 2*F, respectively, before forced cooling was restored. Power to the critical functions monitoring system was lost during the Bus 2A04 outage, rendering some of the temperature indications unavailable, but operators were aware that the calculated time to boil was approximately 28 hour3.240741e-4 days <br />0.00778 hours <br />4.62963e-5 weeks <br />1.0654e-5 months <br /> Technical Specification 3.9.4 requires that one SDC loop be operable and in operation when in Mode 6 with water level greater than or equal to 23 feet over the top of the reactor vessel flange, as were the conditions before this event. When SDC loop requirements were not met, the licensee was required to immediately suspend operations involving a reduction in reactor coolant boron concentration, immediately suspend loading irradiated fuel assemblies in the core, immediately initiate action to satisfy SDC loop requirements, and within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> close all containment penetrations providing direct access from the containment atmosphere to the outside atmospher The licensee completed all these actions, including containment closure, which was no longer required after SDC was restore SDC was restored 26 m'.nutes after it was lost, and the Unusual Event was terminated at 10:40 Conclusions Operators classified the loss of SDC event correctly and promptly. Response to the event was adequate. Operators' execution of the " Loss of Shutdown Cooling" abnormal operating instruction was excellen Operators had not thought through, in advance, the actions that would be necessary to restore SDC in the event of needing to restart the Train A saltwater cooling pum However, the necessary information was available in procedures, and operators worked through the condition expeditiously when it occurred, resulting in only an approximate 10-minute delay in SDC restoration. There was negligible safety consequence to this delay because of the la.ge water inventory. Advance planning for these actions was not required for the circumstances of this even :
The local containment evacuation, although directed by procedure, was a precautionary i and conservative measure. Nevertheless, it was accomplished expeditiously and without hesitatio ,
.
.
-4-02 Operational Status of Facilities and Equipment O2.1 initial System Confiouration Inspection Scoce (37551. 71707)
After Unit 2 experienced a loss of SDC on February 1,1999, the inspectors reviewed operating logs and records of instrument readings of plant parameters, performed inspections of affected plant equipment, and reviewed various licensee procedures addressing shutdown nuclear safety and operation of systems, i Observations and Findinas Decay Heat Removal Systems At the time of the event, a core reload was partially completed, with 177 of 217 assemblies loaded into the core. No fuel movement was actually in progress, and all fuel assemblies in the core and in the SFP were in approved storage locations. The water level was being maintained at greater than 23 feet over the reactor vessel flang The transfer tunnel connecting the refueling cavity with the SFP was open. The high water level established a largc. heat sink for the decay heat from the irradiated fuel assemblie Integrated Operating Instruction SO23-5-1.8.1, ' Shutdown Nuclear Safety," Revision 7, described the requirements for the licensee's defense-in-depth approach to ensuring that shutdown safety functions were fulfilled. Defense-in-Depth Planning Sheet 10A described the primary and backup methods being used to fulfill the shutdown safety functions, including core and SFP decay heat removal, and described other contingency plans or protective measures to be implemented in some circumstance Decay heat removal from the core was being provided by the Train A SDC heat exchanger and Low Pressure Safety injection Pump 2P015. The backup method for decay heat removal relied on a large heat sink, provided by the large water inventory in the refueling cavity. The licensee had calculated that this inventory, with the core reload completed, would result in a 28-hour time-to-boil and a 170-hour time to core uncover The licensee had also developed a contingency plan to restore Train B SDC equipment to service within the time-to-boil margin. Train B was mechanically unavailable primarily because of valve maintenance in progress on the component cooling water syste These methods satisfied the requirements of Integrated Operating Instruction SO23-5-1. Decay heat removal for the SFP was provided by the Train A SDC heat exchanger, the Train A containment spray pump, and the Train A component cooling water system and saltwater cooling system. The backup method of decay heat removal was also the large heat sink. These measures satisfied the requirements of Integrated Operating Instruction SO23-5-1. !
i
.
5 Because the large heat sink was not capable of ultimately preventing core boiling or uncovery, and did not satisfy Technical Specification requirements for SDC system operation, the licensee had developed a formal contingency plan for restoration of the l component cooling water noncritical loop, which would enable the Train B SDC mechanical systems to be operated. The inspectors reviewed the plan, which the licensee had determined could be implemented within approximately 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br />. The plan included reasonable estimates recovery actions, considering the scope of maintenance in progres Electrical Distribution System Train A components were powered from Train A Class 1E 4160 volt Bus 2A0 Bus 2A04 was being backfed from the unit auxiliary transformer, by rneans of a temporary breaker being installed in Location 2A0419, as depicted in Attachment Other available sources of power to Bus 2A04 included the crosstie from Unit 3 4160 volt Class 1E Train A Bus 3A04 and Unit 2 Train A EDG 2G002. The Unit 2 RAT,2XR1, which was the normal power source for Bus 2A04, was deenergized and grounded on the primary windings in preparation for planned maintenance activitie The inspectors reviewed the electrical protection scheme for the RAT and Bus 2A04, including the feeder breakers. Both the undervoltage and overcurrent protection systems were designed consistent with applicable standard The Train B Class 1E 4160 volt bus,2A06, was also available, except that the Train B EDG was not operable. However, because Train B mechanical systems to support SDC and SFP cooling were not available, the Train B electrical availability did not contribute 4 to prevention of, or recovery from, this even I Integrated Operating Instruction SO23-5-1.8.1, Section 6.4, provided direction regarding electrical power availability. Specifically, both Train A Bus 2A04 and Train B Bus 2A06 were considered fully available for fulfilling their shutdown safety functions because of the availability of two independent offsite sources, including two independent circuits from the switchyard (with all four 230 kV bus sections of the switchyard in service) and an EDG. Section 6.4.4 provided guidance for establishing physical barriers to protect power sources "at any time deemed prudent" by Equipment Control or the senior reactor operator Operations supervisor. Such measurcs had not been deemed prudent at the time of the event. Defense-in-Depth Planning Sheet 10A described the above configurations, which satisfied the requirements of Integrated Operating Instruction SO23-5-1. Integrated Operating Instruction SO23-5-1.8.1 did not fully recognize the interrelationships between the electric power availability" safety function and the other safety functions. Specifically, because only the primary method of satisfying the decay heat removal functions (via the Train A SDC and Train A component cooling water systems) provided adequate long-term maintenance of the safety functions, the electric power availability to the Train A systems was more important than it would have been if the Train B decay heat removal systems had been available. Additional measures for
r
.
-6- ;
t i
protecting the Train A electrical distribution systems, such as the physical barriers i described in Section 6.4.4 and Attachment 4 of the procedure, were warranted. This was a weakness of the procedure and had not been identified as a weakness by the licensee's investigation team. The licensee planned to review this consideratio Conclusions The overcurrent and undervoltage protection scheme for the safety-related feeder breakers and Bus 2A04 was acceptabl The defense-in-depth strategy in place for core and SFP decay heat removal provided reasonable assurance that boiling would not occur before a backup active decay heat removal system could be placed in service. The minimal heatup that occurred during the event demonstrated that the large heat sink provided sufficient margin to adequately maintain the decay heat removal safety function for both the core and SF Shutdown Nuclear Safety Procedure SO23-5-1.8.1 was weak in that it did not fully reflect the interrelationship between the electrical distribution safety function and other safety functions that were dependent on the electrical distribution system. Specifically, the availability of Train B Bus 2A06 was credited for fulfilling the electrical distribution safety function as Method B, but did not contribute to supporting the decay heat removal safety functions because the Train B systems were mechanically unavailabl Additional measures to protect the Train A electrical distribution system would have been appropriate but were not required by the procedure. This weakness was not identified by the licensee's investigation and was not included in the corrective action progra O2.2 Eauioment and System Response Inspection Scope (37551. 40500. 71707. 92700. 93702)
After Unit 2 experienced a loss of SDC on February 1,1999, the inspectors responded to the control room and observed operator response; reviewed the causes of the event, including the licensee's event report and action requests documenting aspects of the event; reviewed operating logs and records of instrument readings of plant parameters; performed inspections of affected plant equipment; reviewed the licensee's planned and >
I completed corrective actions; and reviewed the licensee procedure addressing loss of SD Observations and Findinas !
!
TK licensee reviewed the operation of Breaker 2A0418 and determined that the hook ;
on the front of the breaker was an extension of the same piece of metal that was known !
to be the lever that had been pushed. Additionally, the licensee determined that mechanical interlocks do prevent the breaker from closing when racked out, when the i
!
closing springs are discharged by pulling the hook or by pushing the lever as occurred during this event. The breaker functioned as designe .
7 Operators and electricians manually discharged the closing springs on Breaker 2A0418, unintentionally causing the breaker to close and initiating the loss of SDC event. The grounding disconnects on the primary windings of RAT 2XR1 had already been close This configuration resulted in an undervoltage condition on Bus 2A04. Because the ground was on the 30 MVA primary windings and the voltage source for the transformer was on the 10 MVA secondary windings, the transformer acted like a large electrical .
load as the windings energized. The undervoltage protection for Bus 2A04 actuated, I after approximately 1.5 seconds, before an overcurrent condition could develop on the feeder breaker from the unit auxiliary transformer, Breaker 2A0419. The digital fault recorder traces from the event indicated that the sustained degraded voltage circuitry and the undervoltage circuitry had functioned as designed. The licensee's fault current calculations indicated that the fault current did not exceed the overcurrent trip setting for Breaker 2A0419. The inspectors reviewed the fault protection design for the RAT and for the feeder breakers on Bus 2A04 and determined that the design was adequat The ground condition did not cause Breaker 2A0418 to trip free because the fuses had already been removed from the control circuit for the breaker, as was appropriate to ensure personnel safet The loss of voltage signal caused Train A EDG 2G002 to start. Interlocks on the bus prevented the diesel feeder breaker from closing while another feeder breaker was also closed. This protective circuit functioned as designed. Some of the diesel engine auxiliary equipment, such as the radiator fan and the diesel building emergency ventilation system, was powered from a motor control center fed from deenergized Bus 2A04; this auxiliary equipment did not start. Operators stopped EDG 2G002 in accordance with Abnormal Operating Instruction SO23-13-15, Attachment 8. The licensee determined that the engine was protected by the nonemergency engine trip for high jacket water cooling temperature, which would have activated after approximately 20 minutes had operators not acted earlie The licensee developed and completed a thorough inspection and testing plan for potentially affected electrical components, including the RAT and the Bus 2A04 feeder breakers. No equipment degradation was identifie Reactor core and SFP temperatures increased approximately only 5*F and 2 F, respectively, before forced cooling was restored, demonstrating that the decay heat removal safety function was maintained during the even c. Conclusions All equipment and systems, including the undervoltage protection circuitry, performed as designed in response to the event. A thorough inspection and testing plan for potentially affected components, including the transformers and breakers, was developed and completed, revealing no equipment damage as a result of the event. The margin provided by the large heat sink adequately maintained the decay heat removal safety function for the reactor core and SF .
e-8-03 Operations Procedures and Documentation O3.1 Initiatina Actions and Work Controls inspection Scope (37551. 40500. 71707. 92700)
After Unit 2 experienced a loss of SDC on February 1,1999, the inspectors reviewed the causes of the event, including the licensee's event report and action requests documenting aspects of the event; performed inspections of affected plant equipment; reviewed the licensee's planned and completed corrective actions; and reviewed various licensee procedures addressing work controls and operation of system Observations and Findinos The licensee had recently developed and promulgated several initiatives intended to reduce human errors. One of these was to focus on " error-likely situations." These included first time evolutions, high consequence activities, irreversible points of action, inexperience with tasks, high work loads or fatigue, complex problems requiring expert support, hister; cal problem activities, and situations in which personnel have an unclear mente.' picture of the system, process, or scientific principles. Recognition of these kinds of conditions was an important element of the licensee's progra On January 31,1999, the licensee initiated removal of the Unit 2 RATS from service in accordance with Operating Instruction SO23-6-6, " Reserve Auxiliary Transformer !
Operation," Revision 8. Attachment 3 of the procedure required that a switching order be executed to remove the Unit 2 RATS from service and specified that the switching order require checking open the RAT high and low side breakers, including Breaker 2A0418, the feeder to Bus 2A04. After the RATS were deenergized, the switching order required the RAT high side ground disconnects to be closed. These actions were completed by 11 p.m. on January 31. Neither the procedure nor the ,
switching order addressed racking out the low side breakers as part of the switching )
orde The removal of the RATS from service was authorized by Work Authorization Request 2RORT002. Attachment 1 of Operating Instruction SO23-6-6 provided guidance for clearing the RATS and also indicated those actions which would be ,
performed as part of the switching order. Racking out the low side breakers was included in the guidance following the actions performed in the switching orde The result of complying with Operating Instruction SO23-6-6 was that conditions were established in which a single error or fault could occur after the RAT ground disconnects were closed, which could result in the grounded transformer being reenergized from the low side. This established an 'Jnnecessarily risky configuration that was a substantial contributor to both the loss of SDC and the inability of the EDG to perform its design function when it was called upon. This was inappropriate and contrary to the requirements of 10 CFR Part 50, Appendix B, Criterion V, " Instructions, Procedures, and Drawings," which states that " activities affecting quality shall be prescribed by
___
- l l
)
d
l
documented instructions, procedures, or drawings of a type appropriate to the l circumstances . . . ." Operating Instruction SO23-6-6 was designated as quality affecting and was subject to the requirements of Appendix B. This is one example of a Severity Level IV violation (361/99003-01) that is being treated as a noncited violation, consistent with Appendix C of the hRC Enforcement Policy. This example is in the licensee's corrective action program as Action Request 99020003 At approximately 12:30 a.m. on February 1, with the de control power fuses already j removed, operators attempted to rack out Breaker 2A0418 in accordance with Work i Authorization Request 2RORT002. Electricians were present in the switchgear room for )
'
other work. The breaker appeared to jam in the cubicle after a few inches of travel, and the cause was not immediately apparent. Operators made another attempt to rack the breaker out, this time applying so much force that they bent the racking tool. The operators racked the breaker to the full in position and notified their superviso Operators called the Outage Control Center for assistance and received support from i Maintenance and Station Technica I A prejob briefing with electricians and operators was held in the control room, during which a plan was discussed to rack out the breaker and perform a visual inspection. At 2:30 a.m., electricians unsuccessfully attempted to rack out the breaker. A visual inspection revealed three racking mechanism spacer halves that had broken during the earlier attempts. The breaker was left racked in, and Action Request 990200007 and Maintenance Order 990200004 were generate During the turnover for day shift, the problem with Breaker 2A0418 was die ussed by the Electrical Maintenance representatives in the Outage Control Center. i the !
Maintenance shop, the Electrical Maintenance foreman learned of the problem from the I written turnover report from the night shift foremari. The day shift general foreman instructed the foreman to help Operations rack out the breaker using blanket Maintenance Order 9808121600 The problem with the breaker was mentioned at the 6:30 a.m. formal Outage Control Center turnover meeting, attended by managers from most or all of the organizations involved in the outage. Managers did not raise a concern over proceeding with attempts to rack out the breaker and did not recognize this as an " error-likely situation."
At approximately 7:15 a.m., the Operations work process control operator, the day shift )
Electrical foreman, and three electricians discussed removing the breaker. The work process control operator denied a request for verbal authorization to rack out the breaker. The Operations shift manager and the Electrical general foreman were I apprised of the issue. A meeting was held between Operations personnel (including the shift manager, work process supervisor, and two other senior reactor operators) and i electricians (including the foreman). The discussion included potential loss of SDC, the l current breaker status, and precautions to prevent inadvertent breaker closure. A plan was developed to perform a visual inspection and was verbally authorized by Operations. The cause of the racking interference was not identified, although a fourth l half of a racking mechanism spacer was locate '
-
.
.10. .
- Senior managers were again apprised of the problem with the stuck breaker at the 9 a.m. daily managers' meeting. However, the managers did not intervene to require more rigorous review and controls for the efforts to free the breaker The Electrical general foreman and the shift manager each called the Station Technical cognizant engineer, and Operations verbally authorized another visual inspection. The general foreman, three electricians, and the cognizant engineer performed an inspection
. and, at approximately 9.a.m., identified interference between the roller shutter actuator (a pin extending from the side of the breaker) and the shutter lever (part of the cubicle) -
(Attachment 2, Figure 1). They developed a plan to free the breaker; two electricians were sent to fabricate a tool to use to lift the shutter lever and free the roller shutter ,
actuato {
- At approximately 9:15 a.m., a prejob briefing was held in the control room with the shift manager, the common control room supervisor and his trainee, the Electrical general foreman, and the cognizant engineer. The results of the breaker inspection and the plan for breaker removal were discussed. Discharging the closing springs to prevent inadvertent breaker closure, using the lever at the bottom of the breaker, was discussed and included as part of the plan that was approved by this group. During this briefing, the Electrical general foreman assured the group that the breaker would not close when the springs were discharged. The operators determined that this was consistent with i their experience, because they routinely discharged the springs on breakers after racking them out and the breakers had not closed. The shift manager asked about a precaution in Section 4.12 of Operating Instruction SO23-6-2.1, "4160 Volt Air Circuit 4 Breakers," Temporary Change Notice (TCN) 3-1, which stated that pulling the closing spring discharge hook on the front of the breaker would cause the breaker to clos The electricians assured the shift manager that they would not be using the hoo The cognizant engineer went along with the group, who assumed that he understood the functioning of the breaker sufficiently to know if the expectation that the breaker would not close was correct. However, the cognizant engineer did not have this knowledge. Because he did not see his role as providing technical approval of the plan, as'others understood his role to be, he did not object to the plan and also did not say that he lacked the requisite knowledge regarding the breaker. His role had not been explicitly discussed during the briefing. This aspect of the cognizant engineer's performance was weak and resulted in the group unknowingly placing too much reliance on the knowledge of one individual (the Electrical general foreman).
The shift manager directed that another senior reactor operator (the work process supervisor or the common control room supervisor) be present at the breaker when the plan was implemented. A walkdown of the control boards was performed to review for potentialimpac At approximately 9:45 a.m., the work process supervisor, the common control room supervisor, the cognizant engineer, the three electricians, and the Electrical general !
foreman went to the breaker to implement the plan. A prejob briefing was conducted, focusing on industrial safety aspects such as personal protective devices to be use The work process supervisor cautioned against using the hook on the front of the i breaker.-
l l
r
.
Q-11 -
- At 9:59 a.m., an electrician inserted the tool he had made for the task and pushed the lever to discharge the closing springs. (The lever was later determined to be part of the same piece of metal as the hook.) The breaker unexpectedly closed, causing an undervoltage condition on Bus 2A04 that resulted in a loss of SDC. The breaker closure was promptly reported to the control roo The instructions provided to the operators and electricians for freeing Breaker 2A0418, including discharging the closing springs, were inadequate and inappropriate for the circumstances. A blanket maintenance order was used that provided no specific direction regarding breaker operations, the operating instruction for 4160 voit air circuit breakers did not directly address the breaker interlocks, and the verbal guidance was
. incomplete. This is one example of a violation of 10 CFR Part 50, Appendix B, Criterion V, " Instructions, Procedures, and Drawings." This Severitj Level iV violation is being treated as a noncited violation, consistent with Appendix C of the NRC Enforcement Policy (361/99003-01). This example is in the licensee's corrective action program as Action Request 99020003 Conclusions The procedural controls for ensuring that the RAT could not affect operable safety equipment, specified in Operating Instruction SO23-6-6, were inadequate, in that the switching order closed the grounding disconnects before the safety-related low side
~ feeder breakers were racked out. This established an unnecessarily risky configuration that was a substantial contributor to both the loss of SDC and the inability of the EDG to perform its design function when it was called upon. This is one example of a noncited violatio Operators and electricians recognized that the' potential inadvertent closure of the -
breaker would have serious consequences. However, they regarded plans to free the shutter to rack out the breaker as involving low risk and erroneously thought that the plan to discharge the closing springs, intended to prevent inadvertent closure, also involved little risk. None of the licensee personnel involved, including outage management perr'nnel and various supervisors and managers, recognized that attempting to rack out the breaker, once it was known to have interference, substantially increased the risk of the activity and warranted more careful planning and control, consistent with the licensee's recently-promulgated " error-likely situations" guidanc Management missed two significant opportunities to intervene. The programmatic enhancements planned in response to this event are appropriat During prejob briefings discussing the plan to discharge the closing springs to prevent inadvertent closure, undue reliance was placed on the knowledge of one individual. The lack of specific knowledge regarding breaker operational details by others involved, especially the cognizant engineer, was not established or considered. While a
. procedural precaution was discussed, personnel did not confirm that the precaution did not apply, again trusting in one individual's knowledg F'
.,
I*
j -12-l l The instructions provided to the orerators and electricians for freeing Breaker 2A0418, l including discharging the closing springs, were inadequate and inappropriate for the circumstances. A blanket maintenance order was used that provided no specific
- direction regarding breaker operations, and the operating instruction for 4160 volt air circuit breakers did not directly address the breaker interlocks. This is one example of a wncited violatio Quality Assurance in Operations 07.1 Licensee Event investiaation and Corrective Actions a.- Inspection Scope (37551. 40500. 92700)
After Unit 2 experienced a loss of SDC on February 1,1999, the inspectors reviewed the causes of the event, including the licensee's event report and action requests documenting aspects of the event; reviewed the licensee's planned and completed corrective actions; and reviewed the licensee's risk assessment of the even Observations and Findinas
,
Licensee management conducted a stand down of all employees on February 1,1999, i
to inform the employees of the circumstances of the event and to emphasize the
" error-likely situation" wamings that should have been recognized. Additionally, new restrictions were promulgated for control of work for defense-in-depth sheet credited components and systems; no emergent work would be performed except for damage control activities, and safety-related blanket maintenance orders would not be used.
l The licensee initiated a Level 1 event report, the highest level of event investigation in the licensee's system. A team was assembled to conduct the investigation, using
-
,
guidance frorn General Procedure SO123-XV-50, " Event Report Program," TCN 2-1.
l The investigation was chartered to focus on the causal factors of the initiating event, rather than on trying to pursue all performance issues revealed during the event. With
! ' this focus, which was a recent change to the event investigation philosophy, the licensee
! . was able to expeditiously organize, conduct personnel interviews, and gather other
) relevant data'in support of completing the investigation, including identification of l corrective actions and having a draft written report prepared within of a goal of approximately 17 days. Resolution of final comments delayed issuance of the final report until February 25. The report was documented as Assessment SA 99-004 in the
! Level 1 event report,"De-energization of Unit 21E 4kV Bus 2A04."
The corrective action program appropriately focused on causal factors, which were clearly discussed in the Level 1 event report. Corrective actions, including several programmatic enhancements, were clearly identified and entered into the corrective action tracking system (as assignments under Action Request 990200037). However, the event report investigation and report did not identify or address the safety function interrelationship weakness of Integrated Operating Instruction SO23-5-1.8.1, discussed above.
< i
g
.1
a-13-The corrective actions identified in the Level 1 event report included the following:
- -
Develop program controls that set expectations for work control of high consequeryce activities (e.g., a sensitive issues list).
Capture in the lessons leamed briefings the personnel performance issues identified during the event revie *
Capture in the lessons learned briefings the prejob briefing requirements and -
expectation *
' Develop improvements to the " error likely situation" proces *
Revise Operating instruction SO23-6-6 to include racking out low side feeder breakers prior to closing grounding disconnects. (Completed)
- Revise Maintenance Procedure SO123-19.8,"lTE 4.16kV Circuit Breaker-Inspection and Repair," to ensure that shutter roller actuators are installed in the correctlocation. (Completed)
l The Level 1 event report also identified two improvement items related to enhancing training regarding breaker operatio l
'
in a root cause assessment of the stuck breaker, which was also included in the Level 1 event report, the licensee determined that the roller shutter actuator was installed in the incorrect location. The roller shutter actuator was installed in a hole approximately 2.8 inches further back than the hole it should have been in. Because of this error, the roller shutter actuator was inserted in the cubicle beyond a fold in the shutter lever (Attachment 2, Figure 1). The shutter lever dropped slightly, preventing the breaker from being removed until the shutter lever was lifted out of the wa The licensee determined that the 3000 ampere I-T-E Type 5HK350 breaker had recently been overhauled by the breaker vendor and had been installed as Breaker 2A0418 on January 1,1999. The licensee inspected the two other similar breakers that had been overhauled by the vendor, which were in the licensee's warehouse, and determined that all had the incorrect configuration. The correct configuration was not identified in the vendor technical manual or in licensee procedures. The incorrect configuration affected j only the ability to rack the breaker out, which was not a safety function. The licensee's 1 conclusion that the configuration error was the result of a vendor performance deficiency J was rigorously supported, and the licensee event report correctly identified the '
applicability of 10 CFR Part 21. The licensee's corrective actions, including enhancing
. the appropriate circuit breaker inspection procedure, were appropriate. The vendor also informed the licensee that the procedures at the overhaul facility would be similarly '
enhance e
.-
f l -14-l l
The Nuclear Safety Group's risk assess, ment of the event was documented in Report Number NSG-99-001," Risk Assessment of the loss of Train A Bus 2A04 During Fuel Reload, Unit 2 Cycle 10 Refueling Outage." The assessment addressed the relative aspects of the plant configuration and other appropriate modeling assumptions and included the seismic event contribution. The assessment was performed using the licensee's recently upgraded safety monitor. The assessment concluded that the increase in the core damage probability was approximately 4E-9, which was "very small"
-
based on Regulatory Guide 1.174, "An Approach for Using Probabilistic Risk Assessment in Risk-informed Decisions on Plant-Specific Changes to the Licensing Basis." - Conclusions The licensee's Level 1 event report appropriately focused on and identified causal factors of the event. Corrective actions were identified in the report that appropriately addressed identified weaknesses and programmatic enhancements, based on the lessons learned from the event. The inspectors identified a weakness in the licensee's shutdown nuclear safety procedure that was not identified in the event report. The '
.
licensee's investigation and report were completed in a timely manne The root cause assessment of the stuck breaker was good. The incorrect configuration )
affected only the ability to rack the breaker out, which was not a safety function. The licensee's conclusion that the configuration error was the result of a vendor performance deficiency was rigorously supported, and the licensee event report correctly identified
!
the applicability of 10 CFR Part 21. The licensee's corrective actions, including enhancing the appropriate circuit breaker inspection procedure, were appropriat The licensee's assessment of the safety significance of the event considered all relevant factors, correctly concluding that the core damage risk increase was "very small."
08 Miscellaneous Operations issues (92700)
08.1 (Closed) Licensee Event Report 361/1999-001-00: automatic start of an ED l The licensee reported three conditions related to the loss of SDC event o' f )
February 1,1999. These were the automatic start of EDG 2G002, resulting from the loss of voltage on Bus 2A04; the inability of the diesel to perform its design functio resulting from Bus 2A04 being grounded through the RAT; and the improperly configured breaker roll pin, which was reportable pursuant to 10 CFR Part 21. These issues are addressed in Section 01 of this inspection report. This licensee event report is closed.
l 1 l
!
l l
E_
,
e
,
-15-II. Manaaement Meetinas X1 Exit Meeting Summary
!
The inspectors presented the inspection results to members of licensee management at the exit meeting on March 12,1999. The licensee acknowledged the findings presente The inspectors asked the licensee whethor any materials examined during the inspection should be considered proprietary. A vendor drawing of the shutter lever fabrication details was identified as proprietary. Although the drawing had been examined by the inspectors, it was not 1 included or described in this repor !
l I
l l
I
e a
.
Attachment 1 Eequence of Events January 31.1999 Work Authorization Report 2-RORT002 was issued for installation of RAT 2XR3 removable bus links, which required removing RATS 2XR1,2XR2, and 2XR3 from servic :22 The switching order for RAT 2XR1 was issue :00 The RAT 2XR1 switching order was completed, including closure of grounding disconnects. -
February 1.1999 12:30 Operators attempted (first time) to rack out RAT 2XR1 Feeder Breaker 2A041 Electricians were presen :15 Operators, with different electricians present, again attempted to rack out Breaker 2A0418. Excessive force used during the attempt resulted in minor damage to the racking mechanis :30 Operators made several additional attempts to rack out Breaker 2A0418, with the Outage Control Center Maintenance and Station Technical representatives presen :55 Action Request 990200007 was generated with assignment for a maintenance order and an engineering evaluation of caus :00 The Electrical general foreman read a problem with the breaker in turnover lo Electrical supervisors conducted turnover in the Outage Control Cente :00 The Electrical general foreman and supervisors met. The foreman was directed to help operators, using a blanket maintenance order. Outage Control Center managers conducted face-to-face turnover. Operations conducted morning turnover briefing and turnove :30 The breaker problem was discussed at the Outage Control Center shift turnover meetin :00 The general foreman was directed to get the breaker ou :30 The work process control operator told the electrical foreman and electricians not to touch Breaker 2A041 :45 The Operations shift manager, work process supervisor, Unit 3 control room supervisor, control room supervisor trainee, and electricians discussed . making the breaker stable. A visualinspection was planned and the work process control operator verbally authorized the visual inspectio e ,
e
-2-8:15 Electricians visually inspected Bre aker 2A0418. The reason for diffiwlty rack ig the breaker out were not ider*';ted. Broken washers from the racking l mechanism were foun :45 The general foreman and shift manager each called Station Technica :00 The general foreman and Station Technical system engineer arrived at the breaker and observed that the roll pin on the breaker was jammed against the shutter lever, prevanting racking the breaker out. They developed a plan to free the breaker. The nhn did not include discharging the closing springs.-
.
9:00 The breaker problem was discussed at the morning managers' meetin :30 The shift manager, electrician general foreman, Station Technical system engineer, a control room supervisor and control room supervisor trainee decide to discharge the closing springs to reduce the risk of inadvertent breaker closur :45 The electrician general foreman, three electricians, and the Station Technical system engineer conducted a " maintenance tailboard" at the breake :59 Electricians discharged the closing springs of Breaker 2A0418, causing the i breaker to close. The Operations work process supervisor, three electricians, the electrician general foreman, the Unit 3 control room supervisor, the Station Technical system engineer, and an outage management representative were presen l 9:59 .5 seconds after Breaker 2A0418 closed, low bus voltage caused the unit auxiliary transformer feeder breaker,2A0419, to ope :59 All loads were automatically shed from Train A Bus 2A04, except for High Pressure Safety injection Pump 2P017 and 480 voit Load Center 2B0 :59 EDG 2G002 automatically started on loss of voltage signal to Bus 2A04. The output breaker did not close because permissive interlock was not satisfie :59 Core alterations were secure :02 a.m. Operators secured EDG 2G00 :06 a.m. Operators directed precautionary containment evacuation in accordance with the abnormal operating instruction for loss of SD i
l 10:09 a.m. An Unusual Event was declared because of a loss of SDC greater than 10 i
minutes.
I 10:13 a.m. Power to Bus 2A04 was restored from the unit auxiliary transformer. The swing Component Cooling Water Pump 2P025 restarted (breaker was already closed).
.
'
\
c 2 l l
3-10:18 Evacuation of containment was complete :24 Operators restarted Saltwater Cooling Pump 2P307 by positioning the fire isolation switch to the local positio ,
10:25 Operators restarted Low Pressure Safety injection Pump 2P015, restoring SD _
10:26 Operators restarted Containment Spray Pump 2P012, restoring SFP coolin :28 a.m.- Containment clot,ure actions, were completed including containment evacuatio '
i 10:40 The Unusual Event was exite ]
.
i i
F K
C s
Attachment 2 Figures l J
. . J n /
O' 4 NORMAL POSITION (SHUTTER OPEN)
f
. J IA i
AS-FOUND POSITION Figure 1 - Breaker Shutter Lever 220 kv [220 kv cENERATOR O UNrr 2 ,, ,,, ,, 2XR1, GF RMER RVE Isor sus 4kV j ,
'
- 22KV AUXILIARY TRANSFORMER 22KV 22KV 2XU1 AUXL RY 4KV TRANSFORMERS 6.9 KV R2 W
INT 3 Y t;. 2XR3 2Ao4te1 NC No l}No 2A0419 4 4 417 l 3D 2Ao4 ; ,; ;
, , ,
< > < < >
k
< ,/ < r i I t < > f I
d i f l '
U b m
. *N DIESEL GENERATOR 2G002
'
Figure 2 - Electrical Distribution Diagram l
I l
F
".o-
\
-u I'
Attachment 3 Supplementalinformation-PARTIAL LIST OF PERSONS CONTACTED Licensee R. Clark, Quality Manager - Engineering, Nuclear Oversight
~ J. Fee, Manager, Maintenance R. Krieger, Vice President, Nuclear Generation D. Nunn, Vice President, Engineering and Technical Services A. Scherer, Manager, Nuclear Regulatory Affairs T. Vogt, Plant Superintendent, Units 2 and 3 -
R. Waldo, Manager, Operations JtLSPECTION PROCEDURES USED IP 37551: Onsite Engineering IP 40500: Effectiveness of Licensee Controls in identifying, Resolving, and Preventing Problems IP 71707: Plant Operations
,
IP 92700: Onsite Followup of Written Reports of Events at Power Reactor Facilities I IP 93702: Prompt Onsite Response to Events at Operating Power Reactors LIST OF DOCUMENTS REVIEWED Abnormal Operating Instruction SO23-13-1, issue 2, " Local Area Evacuation," Revision 0 Abnormal Operating instruction SO23-13-15, " Loss of Shutdown Cooling," Revision 10 -
Action Request 990200037 Critical Functions Monitoring System printouts for core temperature indications Emergency Plan implementing Procedure Vill-1, " Recognition and Classification of ,
'
Emergencies," Revision 9 Failure Analysis Report 99-007," Preliminary Evaluation of 2A0418 Breaker Ro!!cr issue" ,
!
General Procedure SO123 XV 50," Event Report Program," TCN 2-1 l fnstallation Maintenance Instruction IB 6.2.1.7D, " Medium-Voltage Power Circuit Breakers" Integrated Operating Instruction SO23-5-1.8.1, " Shutdown Nuclear Safety," Revision 7
Maintenance Procedure SO123-I-9.8,"ITE 4.16kV Circuit Breaker Inspection and Repair,"
TCN 2-5 -
i e-2 Maintenance Procedure SO23-19,8.2,"ABB Type HK-350 - 3000 Amp 4kV Breaker Overhaul,"
Revision 0 Maintenance and Surveillance Manual 3.2.1.9D, " Medium-Voltage Switchgear Equipment" Manual "l-T-E/ABB K-Lir.e & HK Breaker Overhaul Procedure Training," dated July 6-17,1998 Operating Instruction SO23-2-8, " Saltwater Cooling System Operation," Revision 19 Operating instruction SO23-6-2," Transferring of 4 kV Buses," Revision 6 Operating Instruction SO23-6-2.1, "4160 Volt Air Circuit Breakers," TCN 3-1 Operating Instruction SO23-6-6, " Reserve Auxiliary Transformer Operation," Revision 8 Operating Instruction SO23-6-6," Reserve Auxiliary Transformer Operation," Temporary Procedure Notice 8-1 Operating instruction SO23-6-9,"6.9 kV,4 kV and 480 V Bus and Feeder Faults," Revision 4 Operating Instruction SO23-6-13, " Procedure for Clearing a SCE 230 kV Line or Bus,"
Temporary Procedure Notice 7-1 Purchase Order No. 6F2T6902, dated October 23,1996 Work Authorization Record 2-R0RT002, "Re-install the 2XR3 Jack Bus Disconnect Links" ITEMS OPENED AND CLOSED Opened and Closed 361/99003-01 NCV electrical alignment sequence inappropriate; instructions for racking out stuck breaker inappropriate (Section 03.1)
Closed 361/1999-001-00 LER automatic start of an EDG (Section 08.1)
LIST OF ACRONYMS USED CFR Code of Federal Regulations EDG cmergency diesel generator NRC Nuclear Regulatory Commission NCV noncited violations RAT reserve auxiliary transformer SDC shutdown cooling SFP spent fuel pool TCN temporary change notice I
L__