NUREG-1190, Transcript of Commission 860122 Briefing in Washington,Dc Re San Onofre & Status of Rancho Seco.Pp 1-105.Supporting Documentation & NUREG-1190, Loss of Power & Water Hammer Event at San Onofre,Unit 1 on 851121 Encl
| ML20198H067 | |
| Person / Time | |
|---|---|
| Site: | San Onofre, Rancho Seco, 05000000 |
| Issue date: | 01/22/1986 |
| From: | NRC COMMISSION (OCM) |
| To: | |
| References | |
| REF-10CFR9.7, RTR-NUREG-1190 NUDOCS 8601300058 | |
| Download: ML20198H067 (500) | |
Text
{{#Wiki_filter:- _ _ _ _ _ _ _ _ - I
~ ~ i
(' ORIGINAL UNITED STATES OF AMERICA i, NUCLEAR REGULATORY COMMISSION In the matter of: COMMISSION MEETING Briefing on San Onofre and Status of Rancho Seco (Public Meeting) Docket No. Location: Washington, D. C. Date: Wednesday, January 22. 1986 Pages: 1 - ins 9601300058 860122 PDR PT9.7 10CFR PDR ANN RILEY & ASSOCIATES , Court Reporters \s__ 1625 I St., N.W. Suite 921 Washington, D.C. 20006 (202) 293-3950
e 1 D I SC LA I M ER 2 3 4 5 6 This is an unofficial transcript of a meeting of the 7 United States Nuclear Regulatory Commission held on S 1/22/86 . in the Commission's office at 1717 H Street, 9 N.W., Washington, D.C. The meeting was open to public 10 attendance and observation. This transcript has not been - 11 reviewed, corrected, or edited, and it may contain 12 inaccuracies. 13 The transcript is intended solely for general 14 informational purposes. As provided by 10 CFR 9.103, it is 15 not part of the formal or informal record of decision of the 16 matters discussed. Expressions of opinion in this transcript 17 do not necessarily reflect final determination or beliefs. No 18 pleading or other paper may be filed with the Commission in 19 any proceeding as the result of or addressed to any stafement 20 or argument contained herein, except as the Commission may 21 authori z e. 22 23 24 25
A 1 UNITED STATES OF AMERICA 2 NUCLEAR REGULATORY COMMISSION 3 - _ _ 4 BRIEFING ON SAN ONOFRE AND STATUS
$ OF HANCHO SECO e _ _ _
? PUBLIC MEETING g _ _ _
9 Nuclear Regulatory Commission J 10 Room 1130 11 1717 "H" Street, N.W. 12 Washington, D.C. 13 14 Wednesday, January 22, 1986 15 16 The Commission met in open session, pursuant to 17 notice, at 10:05 o' clock a.m., NUNZIO J. PALLADINO, Chairman 18 of the Commission, presiding. i 19 COMMISSIONERS PRESENT: 20 NUNZIO J. PALLADINO, Chairman of the Commission 21 THOMAS M. ROBERTS, Member of the Commission 23 JAMES K. ASSELSTINE, Member of the Commission 23 FREDERICK M. BERNTHAL, Member of the Commission 24 LANDO W. ZECH, JR., Member of the Commission 25
0 i j . . i 2 1 1 1 STAFF AND PRESENTERS SEATED AT COMMISSION TABLE: i 1 , 2 H. PLAINE l 3 P. CRANE i 4 V. STELLO
$ T. MARTIN j 6 S. SHOWE i ? J. HELTMES f 8 W. LANNING i 9 W. KENNEDY
! 10 A. SERKiz 1 ! 11 M. CHIRAMAL
! 12 L. PAPAY i
l 13 K. BASKIN 14 ! 15 I ! 16 i ! 17 18 19 20 l 21 i i 22 23 l 24 25
3 1 P RO C E E D I NG S 2 CHAIMMAN PALLADINO: Good morning, ladies and 3 gentlemen. The purpose of today's meeting is for the 4 Commission to receive a briefing by the NRC statt on the 5 status of the San Onofre Unit-1 Nuclear Power Plant. 6 On November 21, 1985 the Southern California Edison 7 San Onotre Unit-1 plant experienced a total loss of inplant AC 8 power as well as a severe incidence of water hammer in the 9 feedwater system which caused a leak, damaged plant equipment 10 and challenged the integrity of the plant's heat sink. 11 Although there were other malfunctions and human 12 errors, plant operators were able to bring the plant to a 13 stable shutdown within three hours also preventing abnormal 14 releases of radioactivity. 15 However, I must say that I was greatly dismayed 16 while reading through the report to learn of the many plant 17 deficiencies, procedural shortcomings and apparent poor 18 licensee approach to root and cause identification disclosed 19 by the investigation of this event. 20 The NRC Incident Investigation Team has evaluated 21 the San Onofre Unit-1 event and today we will hear the team *s 22 description of the event, the principal findings and 23 conclusions. 24 I understand that copies of the team's report,
'45 NUMEG-1190, Loss of Power and Water Hammer Event at San
o
- 4 1 Onofre, Unit-1 on November 21, 1985 is available on the table 2 in the back of the room.
3 During the stafi's presentation of this report, I 4 would like to stati to discuss what follow-up actions are 5 planned by either the stati or Southern California Edison. At 6 the end of today's presentation I would like to ask the
? Commissioners about having Southern California meet with the B Commission at a later date.
9 After our San Onofre discussions, we had planned a 10 short discussion on the status of the Rancho Seco incident 11 investigation and we ought to try to leave at least 15 minutes 12 for that. 13 I understand that representatives of Region V will 14 be listening in on the telephone. 15 Now let me ask it any of my fellow Commissioners 16 have additional remarks at this time? 17 COMMISSIONER ZECM; No. 18 COMMISSIONER ASSELSTINE: Maybe just a comment, 19 Joe. I would agree with your expressions of dismay over the 20 findings of the report in terms of the number and severity of 21 the problems that occurred. 22 I was struck when you look at the number of failures 33 that occurred with the similarities to Davis Bessie and I am 24 very interested in hearing the presentation by the task force. 25 CHAIRMAN PALLADINO: Any other comments?
g 1 (No response.) 2 CHAIRMAN PALLADINO: Then let me turn the meeting 3 over to Mr. Stello. 4 MR. STELLO: Thank you, Mr. Chairman. As you are 5 all aware the San Onofre team is the second one that we have 6 constituted for this purpose.
? CHAIRMAN PALLADINO: Let me ask you a question. Is 8 there any difference between what is on my desk and what we 9 received yesterday?
10 MR. MARTIN: Minor, very minor. 11 CHAIMMAN PALLADINO: All right. Thank you. I have 12 notes on this one. Excuse me, go ahead. 13 MR. STELLO: As you have already noted the third IIT 14 team has already been formed and we are going to talk briefly 15 about that at the end of the meeting today on Rancho Seco. 16 The procedures that we are using and the way the 17 team has gone about it has been evolving with time and we 18 recognize that we are going to need to do more to improve the 19 way in which we go about this activity and we will, of course, 20 be asking Tim when he is finished to give us some more advice 21 and counsel about how we can, in fact, improve our procedures l l 22 and our process. l ! 23 I hope that the licensee, too, from his point of i 24 view will give us any comments that he has in terms of how we 25 can go about doing this job better in the future as I am sure
6 I that this is not going to be the last IIT nor will be the next 2 one will be the last one and we will need to improve that 3 process. 4 You already pointed out that the document that we 6 are going to be discussing that the team provided to me is in 6 the back of the room. I directed the team to put the document 7 in the PDM and to give the licensee of that document. I 8 believe that was done yesterday. 9 The final report will be a NUMEG and it will have 10 NUREG number 1190 and we expect that that will probably be 11 published sometime next week. 12 I would like to take an opportunity to express my 13 thanks to the team because I think they spent many long hours 14 and it was over a holiday season and it does require a great 15 deal et extra effort and I would like to publicly reflect the 16 fact that they and we both know how hard they have to work to 17 get this kind of a job done on a short time scale and in fact, 18 during a holiday season. l 19 COMMISSIONER BERNTHAL: I want to second that. I l 20 have not had a chance to read every last paragraph of this 21 report but I did go through it last night a.little bit and 22 without reaching a final judgment at this point on every 23 detail, it looks like a pretty professional job to me and I 24 want to compliment the statt on carrying on that effort in 25 pretty short order. l
?
1 1 COMMISSIONER ASSELSTINE: I would agree with that. 2 I thought the report was quite well written, very incisive and 3 the recommendations were quite p r.o b i n g . 4 MM. STELLO: Finally, one last point that I want to 5 make, as you recognize the Commission was given a homework 6 assignment over the holidays just as I was. I got the report
? Monday and the rest of the staff, so that is not very much 8 time for us to reflect on the kinds of things that we ought to 9 do and we are not prepared obviously to tell the Commission 10 the kinds of things that we, the staff, ought to do and I am 11 sure the licensee is going to need more time to also respond 12 to what he sees that the team has found.
13 The licensee did indicate right before the start of 14 the meeting, Mr. Chairman, that at the conclusion of the 15 team *s presentation they would like just a few minutes to make 16 a statement and it the Commission would have no objection, I 17 think they are prepared to do that following the team's 18 presentation to the Commission. 19 CHAIRMAN PALLADINO: Would that be any Commission 20 objection? 21 COMMISSIONER ZECH: No. 22 COMMISSIONER ASSELSTINE: I don't have a problem 23 with that. 24 COMMISSIONER ROBERTS: No. 25 COMMISSIONER ASSELSTINE: Although again I agree l 1
1 that we should come back to the question of what we want to do 2 following this meeting. Vic, I think you are right. We have 3 all just gotten this fairly recently. It looks like it 4 deserves a good deal more attention than we have been able to 5 give it in the past two or three days to really digest it and 6 understand the full implications of it. 7 MM. STELLO: I fully concur in that. I do think it 8 would be useful and I think they have heard and can be 9 prepared to come up and make a few comments right after Tim is 10 finished. 11 COMMISSIONER ASSELSTINE: Sure. 12 MR. STELLO: With that, I will just turn the meeting 13 over to Tim Martin and let him present the results of the team 14 effort. 15 MR. MARTIN: Thank you, Vic. First slide, please. 16 (SLIDE.) 17 MR. MARTIN: Mr. Chairman, before we get started I 18 would like to introduce the team members. The first one over 19 on the far left is Matt Chiramal He is our power systems 20 expert and he is from AEOD. Mr. Bill Kennedy, if he would 21 hold up his hand, is from NRH and he is our human factors 22 expert Mr. Wayne Lanning who is the section chief in AEOD 23 responsible for the IIT process was a member of the Davis 24 Bessie team and he provided the continuity to the team to make 25 sure that we were following essentially the same process. Al
9
~
1 Serkis is from NRR. He was a thermal hydraulics expert and 2 Steve Showe is from the NRC training center, PWR's 3 specifically. 'He is a senior reactor operator licensed 4 individual at the Zion plant. The Zion plant happened ti be 5 the simulator that San Onofre uses for San Onofre Unit-1. 6 Next slide, please. 7 (SLIDE.) 8 MR. MARTIN: The following is really some necessary 9 background to understand the event because the plant is 10 uniquely designed. San Onofre Unit-1 is operated by a 11 Southern California Edison. It is located south of Los 12 Angeles near San Clemente, California. it is a Westinghouse 13 three-loop PWR. It was licensed in 1967, 1337 Megawatt 14 thermal, 450 Megawatt electric. 15 It has a spherical steel containment with a concrete 16 enclosure building, a shield building. 17 CHAIRMAN PALLADINO: It is a shield building? 18 MR. MARTIN: It is a shield building but it also 19 provides some way of controlling the release from the l
"O containment. The electric main feedwater pumps also function 01 as safety injection pumps which is unique.
22 (SLIDE.) 23 MR. MARTIN: They have one turbine and one electric 24 auxiliary feedwater pump. Another one will be installed in 25 the near future. They have one immediate and one delayed
l to 1 access olisite power source and they have diesels that start 2 but do not automatically load on just a loss of power. It 3 t a k e.s the. safety injection in conjunction with loss of power 4 to actually load them.
$ COMMISSIONER BERNTHAL: What is a delayed access 6 offsite power source?
? MR. MARTIN: This one where the operators take 8 action to bring power back into the site and we will discuss a 9 little bit about how that works. That is allowed under the 10 GDC-17 criteria.
11 May I have slide 50, please? 12 (SLIDE.) 13 MR. MARTIN: Slide 50 is overview of the plant. The 14 large structure in the middle is the concrete containment 15 building. Just below it is the diesel generators. The things 16 that look like exhaust ports are actually air radiators. This 17 uses radiators for cooling of the water for the pumps. The 18 turbine is the brown structure running right to left. 19 The spider sitting just in iront of the turbine is a 20 flash evaporator. That is going to become important in this 21 event. It will rupture during the event and the auxillary 22 transformers are behind the turbine and a couple of other 23 things. The Pacific Ocean is off the top of the slide. 34 Next slide, please. Would you go to $1, please. 26 (SLIDE.)
11 1 MR. MARTIN: The condensate system has four 2 condensate pumps. Each has a check valve at the end. These 3 are specific MTC check valves. They are made by the same 4 manufacturer of the ones that will fail later on. 5 Coming up both trains, you end up with the flash 6 evaporator in about the middle of the picture which will
? rupture during this event.
8 You will notice that there is nothing that would 9 stop flow from running around this circuit of the feedwater 10 system. They can actually go background. There are no check 11 valves in that line so basically the condensate pumps 12 discharge into a header and then it can go left or right 13 depending upon what restrictions you have. 14 Let*s go to slide 52, please. 15 (SLIDE.) 16 MR. MARTIN: The condensate system discharges into 17 the suction of the feedwater pumps. The feedwater pumps are 18 also safety injection pumps so you will see lines from the 19 safety injection system. The discharge check valves on the 20 feedwater pumps are also going to be very important in this 21 function and you will notice that after you leave the check 22 valve, you again end up into another common header so water 23 can go left or right depending upon what the pressures are in 24 the system. 25 COMMISSIONER ASSELSTINE: Are all these kinds of
12 1 unique features of this particular plant, this combination of 2 the feedwater and safety injection? 3 MR. MARTIN: I personally don't know of another 4 plant that is that way but there may be others. 5 COMMISSIONER ASSELSTINE: All right. 6 MR. MAMTIN: Finally, you get up into the feedwater 7 stations which are on the right hand side of the slide and 8 here again you see check valves. Their purpose is to prevent 9 the AFW, the auxiliary feedwater, which is injected just 10 downstream of them, from coming back into the condensate 11 system and making sure that that AFW goes to the steam 12 generators. Those check valves will also be very important. 13 There are some motor operated isolation valves just 14 upstream of them. They will become important in this event 15 because they will-be the ones that will finally be used to 16 isolate the backward flow. 17 Can we go to slide 53, please? 18 CHAIMMAN PALLADINO: May I ask just one question? 19 Everything to the left of that feedwater pump up there is low 20 pressure? 21 M R ,. MARTIN: Yes, sir, 330 pound tubing. 22 CHAIRMAN PALLADINO: Everything to the right is high 23 pressure? 24 MM. MARTIN: Yes, it is high pressure, 12-something 25 if I remember the pressure.
13 1 CHAIRMAN PALLADINO: All right, 2 (SLIDE.) 3 MR. MARTIN: This is the electrical system. The 4 generator is on the top right coming through a motor operated 5 disconnect, that is a no-load disconnect. There has to be no 6 current going through there when you open that. It then feeds 7 into a common circuit there which goes out through the main 8 transformer to the 220kV yard and comes back through auxiliary 9 transformers into the 1A and IB bus which carry the reactor 10 coolant pumps. 11 CHAIRMAN PALLADINO: Would you identity that as a 12 figure number in the report? 13 MR. MARTIN: That is figure 4.12. 14 CHAIRMAN PALLADINO: Thank you. 15 MR. MARTIN: This is part of the delayed access 16 circuit because once the system is de-energized and you are 17 able to open the motor operated disconnect, then you are able 18 to bring power back through the main transformer to the 19 auxiliary transformers and then into the buses. I 20 That takes a realignment of breakers and finally an 21 operator to close the breaker, therefore, the delayed access 22 feature. 23 Whereas, if you over to the right hand side the what 24 they call auxiliary transformer C teeds the two safety related 25 buses, the 2C and the 1C bus. That is always connected to I
. i 14 1 olisite and there is your immediate access to the olisite 2 power.
3 The safety related buses have non-safety related 4 loads on them. So that is rather unique, too. Each safety 5 related bus is supported by a diesel generator and, of course, 6 the non-safety related loads would be dropped oft in any
? safety injection type of situation.
8 You are able to connett the 1C and the 1A bus 9 together. You are able to connect the IB and the 2C bus , i 10 together. The important thing to recognise is that normally 11 the immediate access circuit is carrying the two safety 12 related buses and the main transformer or the auxiliary 13 transformers associated with the generator are carrying the 14 non-safety related buses. 15 That is the way you would normally operate. 16 Therefore, a trip that would take out the auxiliary 17 transformer C would normally take out both safety related 18 buses. You would therefore lose both feedwater pumps because 19 there is one on each bus and you would not have the event that 20 we had at this time. 21 The peculiarity of the alignment that is going to 22 occur in this event is that they had some grounds and they had 23 connected the IC bus with the 1A bus and had opened the 24 connection between the IC bus and this immediate access 25 circuit, the auxiliary transformer.
15 1 So thres of the buses were being carried by the 2 generator. One of the buses, the 2C bus off to the right 3 hand side, was going to be carried by the auxiliary 4 transformer. 5 So that unusual alignment is one of the things that 6 is going to precipitate the problems. i 7 With that, let's go to slide 54. l
? (SLIDE.) ;
9 MR. MARTIN: This is their main steam system. The 10 thing that I wanted to point out to you here is that there is 11 a common header. 12 CHAIRMAN PALLADINO: What chart would that be? 13 MR. MARTIN: That is 4.1. 14 CHAIRMAN PALLADINO: Thank you. 15 MR. MARTIN: There is a common header that connects 16 the steam space of all three steam generators and there is no 17 way to isolate the steam generators from each other so they 18 can actually feed to a break in any one of them. 19 Further, there is no motor operated mainsteam 20 isolation valve. The mainsteam isolation valves called block 21 valves here have to be manually shut by an operator going up 22 there with a rather large wrench and it is an air operated 23 wrench and shutting it. 24 There are steam dumps to atmosphere. There are also i 25 steam dumps to the condenser. l I
16 1 COMMISSIONER BERNTHAL: How difficult and/or 2 expensive is it to isolate those steam generators from each 3 other? I am surprised. I always thought one of the defenses 4 and redundancies that you had in the case of a loss of coolant 5 accident of some kind is to always be able to dump heat 6 somehow through one or the other of the steam generators. 7 MR. MARTIN: The plant is not designed that way. 8 COMMISSIONER BERNTHAL: Obviously. 9 MR. MARTIN: This plant has been analyzed for these 10 steam break and feedwater break and has been found acceptable. 11 MR. STELLO: He wants to know how expensive it would 12 be to change it. 13 MR. MARTIN: I can't answer the question on how 14 expensive. I don't know. 15 COMMISSIONER BERNTHAL: But it is essentially 16 putting a valve between the three generators. 17 MR. STELLO: If you look at the picture each of the 18 steam generators clearly have to have a mechanism for safety 19 and relief capacity. So it would have to be somehow dividing 20 up the safety and relief system among the three generators and 21 having that capacity. That is required by code. 22 I don't think it is a simple answer and I don't have 23 any idea how expensive it is. You would have to go in and 24 have at least a preliminary design and cost it out. It is not 25 a trivial change.
17 1 MM. MARTIN: It would require additional 2 penetrations and things of that nature. 3 COMMISSIONER BERNTHAL: Obviously our experts have 4 looked at it and I am not a nuclear engineer, but I seem to 5 recall though much being made of the fact in other cases that 6 each steam generator was capable of removing decay heat. 7 CHAIMMAN PALLADINO: That might be a good question 8 for Southern Cal to address not necessarily today but when we 9 meet with them in the future assuming that we will. 10 COMMISSIONER ASSELSTINE: Tim, you mentioned that 11 the analysis had been done to show that notwithstanding that i 12 ditterence in design that the design was still adequate. Did 13 you all independently look at the adequacy of that analysis? 14 MR. MARTIN: No, sir. We did not look at the 15 adequacy of it. We looked to make sure that there was an 16 analysis that showed that. That is all 17 COMMISSIONEN ASSELSTINE: All right. Does anything 18 about this event or your more detailed look at the plant as a 19 result of the event lead you to believe that someone ought to 20 take a look at that issue to determine whether in fact the 21 a.s a l y s i s is correct or not, not just that there was an 22 analysis but in fact, that the conclusions of that analysis 23 were valid or are valid? 24 MM. MARTIN: I personally don't know of anything 25 that would make that analysis questionable in my mind. They
)
18 l I were able to remove heat in this case and shut down the plant 2 safely with an effective feedwater line break. The fact that 3 the other steam generators did steam to the break, that was 4 their heat sink. That is why they were able to dump steam. 5 So it was like a reliet valve 11 nothing else. 6 COMMISSIONER ASSELSTINE: All right. 7 COMMISSIONER ZECH: Before you go on, the bigger 8 picture appears to me to be frankly the fact that we have so 9 many different designs of plants throughout the country and it 10 certainly shows the need for standardization. 11 If you have essentially 55 different utilities with 12 the various combinations of designs, it certainly is clear to 13 me why when we custom build plants like this and many, many 14 others that it makes the utility's analysis as well as our own 15 analysis much more ditticult than it we were standardized. 16 I think that is a larger picture that should be 17 looked into after we check over the real events and the IB significance of this event. It is a clear demonstration 19 certaintv to me of the need for standardization. 20 COMMISSIONER ASSELSTINE: In tact, I gather, Lando, 21 from reading some parts of the report that the differences in 22 design and perhaps weaknesses in understanding et the 23 particular aspects of this design played a part both in our 24 respcase and in the response of the utility. 25 COMMISSIONER ZECM: Yes. I read the same thing. I
19 1 agree that that makes it even more clear. It in our 2 operational center we are trying to figure out this unique 3 design and understand it and all and they are out there doing 4 their best to handle the event while it is occurring, it i d' a 5 very clear message that we indeed need more standardized 6 plants. 7 I agree that it seems to be a real problem in this 8 issue and that is why I etaph a s i z e it at this time. 9 COMMISSIONEN BERNTHAL: There are differences 10 between this one and Zion, I gather, and yet the operators are 11 training on Zion, I can't recall trem my visit there -- 1 12 usually ask whether they are going to build a simulator. Are 13 they building one that is for their plant or not? 14 MR. MARTIN: I can not answer that. 15 COMMISSIONER ASSELSTINE: Maybe we can ask the 16 licensee. 17 MM. MARTIN: Let's move on. Let's go back to slide 18 tour. 19 (SLIDE.) 20 MR. MARTIN: This is an event overview. They are 21 going to have a loss of inplant AC power. They are going to 22 have inoperable feed pump check valves which are going to lead 23 to a rupture of condensate system component, specifically one i 24 of the flash evaporators. They are going to have a complete 25 loss of feedwater for about three minutes.
1
. , 1 I
20 1 They are going to have multiple inoperable feedwater 2 check valves that allow backilow from all three steam 3 generators. They are going to have a water hammer in the B 4 feed line that causes the piping and piping supports and
$ component damage.
6 They are going to damage feedwater check valve which 7 will then develop a significant steam water leak and the 8 operators handled it well and the plant was shut down and 9 cooled down completely. 10 Next slide, please. 11 (SLIDE.) 12 MM. MARTIN. The team was established by the EDO on 13 November 22, 1986 in conformance with the Cemmission approved 14 Incident Investigation Program. Our charter was to determine 16 what happened, to identity the probable tsauses of what 16 happened and then F.ake appropriate findings and conclusions to 17 form a basis for possible follow-on actions. 18 Next slide, please. 19 (SLIDE.) 20 MR. MARTIN: The tact finding methodology was to 21 inteJview people, to meet with people, to review plant data, 22- to review personnel logs, to quarantine equipment that seemed 23 to be involved in the event, to develop with the licensee 24 action plans for the disassembly and inspection of those 25 pieces of equipment, develop an analysis to determine what
21 I happened and then to conclude with some kind of summary 2 report. 3 We made observations of the damaged equipment both 4 in the assembled and disassembled made and we reviewed the 5 licensee *s status reports that they provided us. 6 Next slide.
? (SLIDE.)
8 MR. MARTIN: The sequence of events, the initial 9 condition, they had a small saltwater leak into one of the 10 water boxes in the main condenser. The unit was operating at 11 60-percent power and really searching for this saltwater leak. 12 They had developed a ground on the system, the l '3 electrical system, and they were hunting for it. The steam 14 generator blowdowns were operating at about 100 gpm per steam 15 generator to get rid of the chloride buildup in the steam 16 generators, 17 Their critical function monitor system which is a 18 computer in their tech support center was disabled. Part of 19 the troubleshooting activities, they killed power to it 20 momentarily and had not reset it. As a result, all of the 21 data from it was not usable to us or to them. 22 The electrical plant was in the unusual alignment 23 that I had previously talked about. 24 Next slide, please. l 25 COMMISSIONEM ZECH: Before you go on to that, as I
22 I tried to go through the diagram of the electrical circuits 2 last night and it seemed to me that they were not on the usual 3 alignment because they were trying to find the ground, is that 4 right? 5 MR. MANTIN: They had found the ground and were 6 isolating that ground from the safety related bus. 7 COMMISSIONER ZECH: Then I couldn't figure out why 8 they did that. It seemed to me that was perhaps an incorrect 9 alignment because they put as I understand the diagram, they 10 put the grounded circuit on one of their vital buses. Is that 11 right? 12 MM. MARTIN: The actual ground appeared initially on 13 their safety related bus. They have a meter that is on the 14 safety related bus and it went to 100-percent and there was'a 16 ground. 16 COMMISSIONER ZECH: Then why did they go to this 17 unusual alignment? 18 MR. MARTIN: In their investigation in trying to 19 find the ground, they were taking loads ott and putting them 20 back on and watching to see 11 the meter cleared. Finally, 21 they got the point where they had a non-safety related bus 22 parallelled with the safety related bus and they got the idea, 23 now that we have power to the safety related bus, let's open 24 the normal feed from the auxiliary transformer and see it it 25 still stays there.
23 1 They opened it and the ground went away. Th.at told 2 them it was not on the safety related bus, it was on the 3 transformer. 4 COMMISSIONER ZECH: Yes. 5 MR. MARTIN: That was how they got into the 6 alignment. Then they started to do some further 7 troubleshooting trying to localize that ground so that they 8 could isolate it and take it out so that it would not affect 9 a safety related piece of equipment. 10 COMMISSIONER ZECH: Was your conclusion that those 11 were all correct steps to take? 12 MR. MARTIN: They were not per procedure and the way 13 they did it jeopardized other equipment. 14 COMMISSIONER ZECH: Yes, that is why I was asking. 15 MR. MARTIN: The 1A bus which is not safety related 16 is fed by a Y transformer which is an installed ground in it. 1 17 COMMISSIONER ZECH: Right. f 18 MR. MARTIN: Installing that ground with a system 19 you know already has a fault will generate circulating 20 currents that will further degrade the insulation in the 21 system. We can't prove a cause and effect relationship that [ ( 22 the event occurred today because such-and-such but that is not 23 wise to do that. 24 COMMISSIONER ZECH: Yes. That is kind of what I l 25 thought, too. It wasn't clear to me that they really had not l l
24 I thought that through clearly and that they might have 2 anticipated further problems by energizing that bus that was 3 grounded. 4 MR. MARTIN: We certainly have Monday morning 5 quarterbacking. ~ 6 COMMISSIONER ZECH: As long as you have looked at 7 that, I think it is important and they look at it, too, 8 because it they did something incorrectly, it seems to me that 9 they should know about it. 10 MR. MARTIN: We felt strongly enough to point that 11 out, 12 COMMISSIONER ZECH: All right. 13 CHAIRMAN PALLADINO: But I understand they do have 14 procedures which they didn't follow for some reason. 15 MR. MARTIN: Actually and this is really jumping 16 ahead to one of the conclusions, as we got down to that point 17 every time they got to a step that would take them into an 18 action statement, de-energizing the feed pump which happens to 19 be a safety injection pump, would have taken them into an I 20 action statement De-energizing the IC bus which is part of 21 the procedure would have taken them into an action statement. 22 De-energizing the auxiliary transformer would have taken them 23 into an action statement. 24 So every time they seemed to get to that point they 25 then considered does the procedure cover all the things that
25 1 might have been giving us the ground that wouldn*t take us 2 into an action statement. 3 They looked further. Th.ere was no liability to 4 going into the action statement and coming back cut of it. 5 The turning off the pump and then putting it back on was a 6 simple act. Yes, it would take you into the action statement 7 but action statements are there for that purpose. They are 8 there for surveillance. They are there for preventive 9 maintenance. They are there for troubleshooting. 10 You can go into them. If you find the problem and 11 fix it, you can come back out of them. 12 COMMISSIONER ZECH: It looks to me like they were 13 trying to avoid the tech spec action statements. I was going 14 to ask ycu why were they doing that? 15 MR. MARTIN: It looks like they have a very strong 16 respect for the tech specs and
- Y. s y
. on* t like to get into a 17 condition which violates -- l i .s e limi ting conditions for 18 operations. If you are in a condition that violates your 19 limiting condition of operation, you have an action statement.
20 It says either repair the equipment or shut the 21 plant down in a very short period of time. It gives you a 22 time when to do it 23 COMMISSIONER ZECH: Those are put in there on 24 purpose I think with a lot of thought It seems to me that 25 those are safety related steps that have been thought through _-r _ _ - - _
26 1 and should be followed. I couldn*t really see why they were 2 avoiding that and getting in that condition. Would you agree 3 with that? 4 MM. MARTIN: We reached the same conclusion. We
$ didn*t see why they didn*t go into the action ~ statement and 6 follow their procedures.
7 COMMISSIONER ZECH: Why didn*t they? Perhaps they 8 should address it but it seems to me that it might have been 9 something that they were intentionally training people to 10 avoid. That would concern me a bit. Maybe that is not true 11 but it seems to me from what I heard from your report, that is 12 the case. 13 MR. MARTIN: That was our concern and we did not go 14 far enough to tell you what was the actual cause or why they 15 did the things they did. 16 COMMISSIONER ZECH: All right, but you have pointed 17 out and perhaps we should hear from them as to their views on 18 how they look at responding to this kind of an event under 19 these circumstances. Would you agree with that? 20 MM. MARTIN: I agree, sir. 21 COMMISSIONER ZECH: All right. 22 (SLIDE.) 23 MR. MARTIN: Into the sequence of events, slide 24 nine, at time zero the auxiliary transformer C isolated as a i 25 result of its protective circuitry sensing a ground fault in
27 1 excess of 1,500. amps. The ENS phone rang almost immediately. 2 The feedwater pump check valve failed on the pump that was l l 3 coasting down. 4 As a result, the west feedwater pump was still 5 running and it was able to pressuriz'e the east feedwater train 6 and it ruptured the flash evaporator tubes which we previously 7 told you were about 350 pound tubes. 8 CHAIRMAN PALLADINO: Was there any finding on why 9 the ENS phone rang? 10 MR. MARTIN: We tried to duplicate that and could 11 not, Just de-energizing won't do it. Noise on the circuit 12 might. But we were not able to duplicate it although we have 13 multiple confirmatory reports that it occurred. 14 Further, when the phone was picked up at both ends 15 each thought the other had called them. The shift supervisor 16 who picked it up couldn*t figure out how NRC was so perceptive 17 that they knew exactly -- 18 (Laughter.) 19 CHAIRMAN PALLADINO: All right. Thank you. 20 MM. MARTIN: The diesels did start on the bus that 21 was de-energized. The operators then very quickly following 22 their procedures, they looked up, they saw that they had had a 23 differential trip. They looked over the buses and there is a 24 little bunch of lights but they immediately saw it was vital 25 bus 4 and they have a procedure that says when vital bus 4
28 1 goes, you have to scram the plant. So within 20 seconds these 2- operators had jumped to their panels and had hit the trip 3 switches. That then led to a total loss of power because you 4 lost your immediate access to offsite and you just tripped 5 your main turbine generator so all power was lost to the site 6 except that led by inverters which were DC power. 7 CHAIRMAN PALLADINO: The diesels had not been loaded 8 at this time? 9 MR. MARTIN: No, sir. Then the second diesel now 10 starts because of the rest of the loss of power but it doesn't 11 load either because that is not the way the thing is designed. 12 COMMISSIONER ZECH: Could you tell me why is it 13 designed that way? Why doesn't it pick up the load on loss of 14 power? 15 MR. MANTIN: It is designed to do that because the 16 preferred source at this site is offsite. Our General Design 17 Criteria number 17 allows this way to deal with it. Beyond 18 that in the review, I can't answer. 19 COMMISSIONER ZECH: Most plants as I understand it 20 though are not designed that way. l l 21 MR. MARTIN: That is also my understanding. l 22 COMMISSIONER ZECH: So this is something that ought 23 to be looked into, also, it seems to me. Why don't they pick 24 up? That is what they are for. 25 MR. MARTIN: I think that a comment needs to be made j
29 I here that the team was specifically told that we will not 2 re-review the design of the plant. 3 COMMISSIONER ZECH: I understand that because I had 4 looked at this whole situation and I couldn't help but think 5 that perhaps the design should be looked into. It certainly 6 contributed to in my view anyway at this stage, contributed to 7 the event. So the design perhaps is adequate but certainly I 8 think there were enough things that happened that we.should 9 perhaps take a look at the design. 10 It seems to me that this is rather apparent and 11 something we should do. 12 COMMISSIONER BERNTHAL: I think the design was 13 looked at perhaps not in certain of those detailed areas, but 14 it was certainly looked at pretty carefully prior to the 15 Commission's start-up decision on that plant. When was that 16 now, sometime ago or at least the modifications that were 17 agreed to by the utility and I don't know whether Harold wants 18 to report again on how pervasive that look was. 19 COMMISSIONER ZECH: No question about it but I am 20 just saying that now perhaps this event leads us to take 21 another look 22 COMMISSIONER BERNTHAL: It might be worth hearing 23 from Harold j u s.t how pervasive and complete and thorough their 24 check was or whether it focused on the immediate questions 25 that were before the Commission at that time.
30 1 COMMISSIONER ASSELSTINE: My recollection was though 2 that review however pervasive it was really focused on the 3 seismic design adequacy rather than on these kinds of design 4 questions on the plant.
~
5 COMMISSIONER BERNTHAL: Probably. That may be true. 6 CHAIRMAN PALLADINO: I think Southern Cal ought to 7 take note of the question because we would be interested in 8 why they designed it this way. 9 COMMISSIONEM ASSELSTINE: Is there any indication 10 that this design aspect as well as a couple of the others that 11 you have mentioned are characteristic of the vintage of the 12 plant or not? 13 MR. MARTIN: I acknowledge that this is an old 14 plant. I can't comment on whether it is characteristic. 15 COMMISSIONER ASSELSTINE: All right. One question 16 it seems to me is to what extent these kinds of more manual 17 functions or operations are characteristic of the fact that 18 this is one of the earlier plants. This is our DC-3. 19 COMMISSIONER BERNTHAL: Unfortunately I think manual 20 functions and operations is a philosophy that until very 21 recently has been pervasive in all plants even some of the 22 plants or most of the plants that are coming on line today. 23 I have argued for a long time that is the wrong 24 philosophy. One simply can't run things that way any more in i 25 this day and age and NASA learned that a long time ago. I
31 I think the industry has learned it as well or the vendors have 2 learned it. It is just that the progress at the vendor level 3 and progress was made long ago in European controlled design 4 and it has just not appeared yet in this country. 5 MR, STELLO: Let me comment. You recognize we, too, 6 have many of the questions that you are raising that we want 7 to look at and we will I am sure the licensee will, too. 8 Without getting into a lot of detail, it 9 important to recognize that these diesels would have loaded it 10 there was a fast response needed, that is if they had a safety 11 injection need, then the diesels would have been picked up. 12 That is when they are needed quickly. 13 Without trying to decide which way it ought to come 14 out, you recognize that the preferred source ought to be ) 15 offsite power. So much more equipment can be run in the plant 16 when the olisite power system is brought back in and that is i j 17 one thing that one ought to do quickly because the diesels 18 clearly can't pick up all of the loads in the plant and run 19 all of the equipment in the plant. 20 I don't want to pre-judge how it comes out but I 21 would at least point out that there are some reasons not 22 trivial for suggesting that that aspect of the design has some 2 '3 merit, too. What we need to do now is systematically go back 24 and look at both sides of the issue and try to come down, is 25 there a reason to decide for change.
32 1 I wanted to point out that we need to do that 2 carefully because there are good reasons for keeping some 3 things the way they are. 4 CHAIRMAN PALLADINO: Is this part of your tollow-on 5 action? 6 MR. STELLO: That will be part of what we will look 7 at as our follow-on and we haven't done that yet. 8 CHAIRMAN PALLADINO: Good. 9 COMMISSIONER ZECH: My only point is that it is 10 certainly timely. I am sure that the analysis proved that it 11 was a proper way to operate the plant. I have no doubt about 12 that but it does seem to me that since this happened, it is 13 certainly timely for us to look at it again and see whether we 14 at this stage in 1985 have the same conclusion. 15 MR. STELLO: Sure. 16 COMMISSIONER ZECH: Perhaps some improvements should 17 be made and perhaps not but 1 just think that it should be 18 looked into. 19 MR. STELLO: I agree with you. 20 COMMISSIONER ASSELSTINE: Do we know how long the 21 diesels could run unloaded without the benefit of some of the 22 cooling systems that are operated by AC power? 23 MR. MARTIN: If the safety related buses were not 24 powered within 39 minutes the diesel will begin to overheat 25 because of the AC radiator fans would not start.
33 1 COMMISSIONER ASSELSTINE: All right. 2 MR. MARTIN: That used to be in their procedures. 3 It has gotten dropped. I think it is generally known in their 4 training but the procedures don't cover it.
$ COMMISSIONER ASSELSTINE: All right.
6 (SLIDE.) 7 MR. MARTIN: At this point we have had a complete 8 loss of power. We have had complete loss of feedwater. The 9 levels in the steam generators dropped. The turbine-driven 10 auxiliary feed pump starts but it will be in a three and a 11 half minute start-up period to warm up. 12 Four additional feedwater check valves failed at 13 this time, also and as a result the feedwater lines began to 14 empty as the steam generator pressure blows that water back 15 toward the condensate system which has the failed flash 16 evaporator. 17 The loss of voltage automatic transfer scheme starts 18 now because it is trying to set up for this delayed return of 19 power from oitsite and the fire truck arrives on site as a 20 result of communications between a fire watch and the 21 misconception that the steam might be smoke. 22 Next slide, please. 23 (SLIDE.) 24 MR. MARTIN: The operators then try to restore power 25 when they don't think the auto sequence has succeeded. They
34 I waited the two minutes they thought it was going to take and 2 they don *t get the end of sequence light so they start to try 3 to bring the power back themselves. 4 The turbine-driven feedwater pump has now come up to 5 speed. It is starting to deliver water to the lines but the 6 check valves are still stuck open and so that water instead of
? going to the steam generators goes right toward the condensate 8 system.
9 The operators have a great deal of difficulty 10 bringing the power back in. They don't normally operate the 11 220kV circuit breakers and as a result, it takes them five 12 times before they succeed in doing that. 13 The motor-driven auxiliary feedwater pump starts 14 once power has been restored. 15 CHAIRMAN PALLADINO: Why did it take five times to 16 restore the power? 17 MR. MARTIN: They just don't have experience with 18 these and there is a series of interlocks that you have to 19 press. You have to move a synchroscope knob down into the 20 right slot, turn it on. You then have to have reset the 21 lock-up bus on another panel and then you have to hold a 22 by-pass switch button while you are turning -- 23 CHAIRMAN PALLADINO: You have convinced me. 24 (Laughter.) 25 MR. MARTIN: Once the power was restored now we are
. l o .
l 35 i I about tour or five minutes into the event, the operators are 2 back into their procedures for the reactor trip. The 3 operators closed the motor-operated valves that I pointed out 4 to you in the feedwater system and that stops the blowdown of 5 the steam generators, the emptying of the feedwater lines and 6 the lines start to fill up again. 7 This is going to set up the conditions for the water 8 hammer that is going to occur later. 9 COMMISSIONER ASSELSTINE: Up until that point, did 10 they have any heat sink essentially? 11 MR. MARTIN: Yes. It was all going out the break. 12 It was going out through the feedwater lines and head toward 13 the condensate system. 14 COMMISSIONER ASSELSTINE: All right, 15 MR. MANTIN: The steam generators were steaming down 16 during this period of time. The steam water level was still 17 visible in the wide range at this point. 18 The operators then reset the radiation monitor 19 alarms unknowingly resetting the steam generator blowdowns. 20 So now you have an additional 100 gpm coming out of the bottom 21 of each one of the generators. That is going to lead to the 22 level dropping a lot faster and actually losing the indication 23 of level later on. 24 Next slide, please. 25 (SLIDE.)
36 1 MR. MARTIN: the operators are not looking to their 2 panels. They see that the NCS pressure is dropping, 3 temperatures dropping, pressurized levels dropping, the 4 charging pumps have been lost on loss of power so they start 5 the charging pumps. 6 Once they got the charging pumps started -- 7 CHAIHMAN PALLADINO: Did they have power restored at . B this time? 9 MR. MARTIN: Yes, sir. 10 CHAIRMAN PALLADINO: Yes, I remember, the fifth 11 time. 12 MR. MAMTIN: Once they got the charging pumps 13 started they noticed that the auxiliary feedwater was going 14 also and they got concerned that maybe that was causing the 15 rapid cooldown and loss in level. So shortly after starting 16 the B reactor coolant pump to get out of natural circulation 17 which was working at this point to get back to forced 18 circulation, the operators then terminated auxiliary 19 feedwater. 20 The way the control room was operated, the operators 21 are at the panels very enmeshed in what is going on at their 22 panels. Another man is reading the procedures out loud. He 23 says, "Have you started this?" "Have you done that? He gets 24 responses. 25 But the shift supervisor is standing back. He is
3? j I watching what is going on, the big picture. He sees the man 2 responsible for the primary system doing the right thing for 3 his primary system. He has told this guy on the secondary 4 system to stop feeding the steam generators. The shift 5 supervisor doesn*t want the generators to go dry so he 6 requires them to be reinitiated about 25 gpm, a lot lower flow 7 rate but hopefully enough to keep them from being dry. 8 The operators then operate like this for a while. 9 They then declare an unusual event on site and somehow through 10 some oversight that information never got to NRC. 11 COMMISSIONER ASSELSTINE: I take it, Tim, from wnat 12 you described the shift was functioning as it was intended? 13 MR. MARTIN: It was a beautiful team work job. The 14 way the STA and the shift supervisor and the control room 15 supervisor and the control operators worked together is 16 laudatory, it really is. 17 COMMISSIONER ASSELSTINE: Good. So what we wanted 18 to accomplish by having the second SHO is exactly what was 19 taking place. The shift supervisor was able to step back. 20 MR. MARTIN: Yes, sir. I think that was one of the 21 very positive aspects. 22 COMMISSIONER ZECH: It seemed like that STA combined 23 SHO, in particular, did a very line Joh. 24 MR. MARTIN: He did. In fact, he was basically 25 sensitizing the shift supervisor, "You know, we have a low
38 l 1 level, be conscious of that. It is not a red alert. It is a 2 yellow alert but be conscious of it " He kept coming back 3 just reminding, "You still have that thing to take care of " 4 He was the one that actually found later on the 5 steam generator blowdowns were open and got them to shut it 6 off So the system worked well in terms of people in the 7 control room. 8 (SLIDE.) 9 MR. MARTIN: As the lines tilled up, the A and C 10 lines tilled up fairly quickly because they are about halt 11 the length of the E lines. The B line was about 90-percent or 12 more full when the water hammer occurred and we will talk 13 about that later. 14 Shortly thereafter they got a high temperature alarm 15 on the B reactor coolant pump. They looked at their meters. 16 They saw it was tracking along and then suddenly went 17 off-scale. They felt it was an inaccurate reading, that it 18 was probably a sensor failure but they didn't want to take a 19 chance. They decided that they were going to start the other 20 pumps and shut this one down. 21 The operators secured the diesel generators and they 22 later start the two A and C reactor coolant pumps. The level 23 has been steaming down in the steam generators as a result of 24 the blowdown. 25 Can we have the next slide, please?
{ 39
~
1 (SLIDE.) 2 MR. MARTIN: At this point the reactor coolant pump 3 A and C are started. The B reactor coolant pump has just 4 stopped. The steam generator water level has just gone 5 off-scale on all three steam generators and the shift 6 supervisor directs that they can increase the auxiliary
? teedwater flow to the A and C steam generators because those 8 are the ones that have reactor coolant pumps running.
9 He didn't know that there was something wrong with 10 the B. In fact, when the water hammer occurred the B teed 11 line had ruptured, not ruptured, but the bonnet on a check 12 valve had been lifted, it stretched the bolts about a half an 13 inch and they got a big steas sater leak going on there. 14 They are going to learn about that very shortly 15 because the operator who was down there shutting the main 16 steam stops comes up and he is just soaking wet and tells them 17 that there has been this steam line break. They go right 18 together, side-by-side. 19 The steam generator blowdown is discovered and I 20 talked about the STA process ict that. The steam generator A 21 and C levels then come back on the scale because you are 22 feeding them at the higher rate. The operators are trying to 23 establish additional cooling in the plant They have problems 24 regarding the circulating water pumps and it is interesting 25 but not very significant.
40 1 They then enter mode tour as their cooling down. As 2 they are getting down to the residual heat removal system they 3 get down to about 400 pounds, go on down to about 370 and that 4 should have been the point where you were at low enough 5 pressure to start the NHH system and they try to open the 6 valves and they won't open.
? It is a problem that they didn't know where the 8 actual interlock set point was. The circuitry worked fine.
9 It was just their knowledge and their training and their 10 procedures didn't really cue them to what the real set point 11 was going in the decreasing pressure direction. 12 COMMISSIONER ASSELSTINE: It wasn't in the 13 procedures? 14 MR. MARTIN: It talked about a 400 pound nominal set 15 point. In actuality, you have to get down to 367 pounds going 16 down and then as you go up, the interlock will clear at about 17 397 so it is nominally 400 pounds is what they understood and 18 their procedures didn't really cue them. You had to get down 19 to like 350 pounds before you would be confident that the 20 interlock was cleared. 21 COMMISSIONER ASSELSTINE: all right. 22 MR. MARTIN: Once they looked at the pressure though 23 they knew the system was not going to be over pressurized 11 24 they did open those. They override the interlocks thinking 25 the interlock had tailed and in fact, it had not.
l l 41 I l 1 Subsequently, the unusual event was terminated. The 2 feedwater leak was isolated and they entered mode five early 3 for refueling outage. 4 (SLIDE.)
$ MR. MhMTIN: In debriefing the operators, they 6 learned of the water hammer and recognized that they had to 7 inspect the lines to see 11 there was any problem. That night 8 they entered the containment and they found the damage on the 9 feedwater line.
10 That is really the sequence. May I have the next 11 slide, please? 12 (SLIDE.) 13 MR. MARTIN: In terms of personnel performance 14 evaluation, I kind of dwell on the bad things here but I have 15 a lot of positive things to say about these operators. Within 16 20 seconds they scrammed the plant, 17 They were conscious of the primary system, the 18 secondary system and they were most concerned with the 19 primary system. That was good. 20 They used procedures very well once they were into 21 the event. The team work was excellent and they safely shut 22 down the plant. 23 But we also found some errors. Most of these errors 24 are associated with early on in the way they followed the 25 procedures for troubleshooting the electric plant. We feel
42 1 the things they did were nct appropriate. 2 They did have ditticulty in re-establishing inplant 3 power. That was a training problem and tailure to follow some 4 specific procedures there. They inadvertently re-established 5 the steam generator blowdown. Pert of that is because you 6 ' have no indication of blowdown in the control room so you
? don't know that you did.
8 That is a design deficiency as far as far as we are 9 concerned and they didn't reset this POX III Computer when 10 they were doing the troubleshooting early. They didn*t reset 11 it after the plant lost power so in both cases most of the 12 digital data, all of the digital data, was not available to us 13 and was not available to them to try to understand where they 14 were either. 15 It they had gone into an alert and activated the 16 tech support center, those people would not have known how 17 they got to where they were. 18 The STA performance was good. The emergency 19 coordinator which is an E-Plan requirement, his performance, 20 he had some problems. 21 CHAIRMAN PALLADINO: Can I ask you about the 22 computer again? When did it go out? 23 MR. MARTIN: It went out first when they were doing 24 the troubleshooting for the ground and were de-energizing the 25 parts and they didn*t reset it then. Then later on when they
I I 43 1 had a loss of power it boots up, it starts up, but you have to 2 reset it because of the software and the way it interacts. 3 CHAIRMAN PALLADINO: Is that something ditticult to 4 do? 5 MR. MARTIN: My understanding is that it is very 6 easy. 7 COMMISSIONER BERNTHAL: You just press a button. 8 MR. MARTIN: It is just something you have to 9 remember to do, that is all, and there was NRC-licensee 10 communication problems. 11 Next slide, please. 12 COMMISSIONER ASSELSTINE: Did they have an SFOS or 13 anything else that would give them the plant data? 14 MR. MARTIN: No, sir. 15 COMMISSIONER ASSELSTINE: They don't have an SPDS? 16 MR. MARTIN: No, sir. 17 (SLIDE.) 18 MR. MARTIN: The power supply cable that ran from 19 the C transformer to the 1C bus had a ground fault in it. 20 There were two holes in it. It appears that there may have 21 been some water inleakage into the armoring around the cable 22 and the analysis is ongoing for that. 23 The flash evaporator unit, the tube ruptured inside 24 and then that shell is only a 15-pound shell and of course, it 25 ruptured fairly quickly after that. The flash evaporators are
44 1 no longer used except that the condensate system still has a 2 path which goes through the condenser side of that 3 evaporator. So they are able to cut away all the things 4 except that condenser portion. Whether they will replace that
$ with a solid pipe or not, that is their call.
6 The safety injection annunciator, they got a 7 spurious alarm of a safety injection but two days earlier they 8 had gotten one when they were maneuvering buses around and so 9 they knew it was probably a spurious alarm. It turns out that 10 overy time you lose power at this plant, you are going to get 11 that alarm. 12 COMMISSIONER ASSELSTINE: Did they know that or not? 13 MM. MANTIN: Not at the time. They knew'that they 14 had experienced this two days before so there was a certain 15 group of operators that knew that but it wasn't in their 16 procedures. It didn't say, "Oh, by the way, you can*t trust 17 this alarm when you have a loss of power." 18 Complicating that, they have some remote safeguard 19 load sequencing panels which are located in the control room 20 and they also indicated that they had had a safety injection 21 actuation. We have yet not been able to find out why that is 22 and the licensee is still evaluating that. 23 COMMISSIONER ASSELSTINE: I take it the operators 24 correctly diagnosed that in fact -- 25 MM. MARTIN: What they did, was they acknowledge
G5 1 both of them and then they looked at their plant parameters 2 and they saw that the plant parameters didn't call for one. 3 They did have electric power back so they wouldn't have seen 4 any pumps or anything running, but they made the determination 5 that H e y , it is a spurious alarm." There is no r+ason, the 6 contali, ment pressure is not up, the pressure in the RCS is not 7 low enough. There is no reason for this system and they went 8 right on beyond that. They did right. 9 The loss of voltage auto transfer scheme, it failed 10 to realign the circuit breakers to restore the power but the 11 reason has not yet been identitled. We did not want to do any 12 testing that would compromise the source of power to this 13 site. 14 The C transformer was not available through most of 15 the investigation. One of the diesels had been taken apart 16 for maintenance so it was just not a good idea to finish the 17 investigation in this area until we had reliable second 18 offsite power. 19 We just basically said that we will wait until that 20 is done before you do the test. 21 CHAIMMAN PALLADINO: tim, in the text I get the 22 impression that root causes will not be developed in many 23 circumstances except for perhaps the feedwater system check 1 24 valves. Yet this slide and several others indicate ongoing I 25 analysis. Did I get the wrong impression from the text or how l
=
40 1 do you reconcile these statements and what I got trom it? 2 MH. MANTIN: The licensee is continuing his analysis 3 and in fact, he will develop a report before he starts up 4 showing what the trip occurred, why it occurred, here are the 5 problems and here are his fixes. That is an ongoing process 6 but the investigation, the NHC investigation, has run its
? course, 8 CHAIRMAN PALLADINO: You say on page 1-3, "However, 9 with the possible exception of the identification of the root 10 cause of the feedwater system check valve failures, it is 11 unlikely that new significant information relative to what 12 happened and why will be developed "
13 MH. MARTIN: The key 'there is significant. A lot of 14 these problems -- 15 CHAIRMAN PALLADINO: Is it unlikely because you 16 can't the data or is it unlikely because nobody is going to 17 plan to do something? That is what I was trying to 18 understand. 19 MH. MARTIN: It is because we are a little bit cooky 20 and we think we understand it to be quite trank and what we 21 put in our report and what we say is the probable cause we 22 believe very firmly is the probable cause. 23 CHAIHMAN PALLADINO: But here you have indications 24 that you do have or Southern Cal has analyses ongoing that may 25 disclose root cause,
47 1 MR. MARTIN: That is correct. l l 2 (SLIDE.) 3 Mk. MAMTIN: The problem with the FOX III computer, 4 we talked about the power interruption. Some turbine disks 5 did rupture. That is expected on a loss of power. We have a j 6 load reject and you have all the pressure still inside the 7 turbine, that is what they are there for. They did what they 8 were suppose to. l 9 COMMISSIONER BERNTHAL: It has to be a trivial task l 10 to set up a computer system, a data system like that, to reset i 11 itself when power is restored and boot itself up. Why isn*t 12 that done? 13 MR. MANTIN: I can't answer that. 14 CHAIRMAN PALLADINO: That is a good question for 15 Southern Cal. 16 MR. MANTIN: The emergency notification system, 17 there was spurious ringing. We have not been able to le reproduce the conditions that caused it although we did 19 extensive investigation in that area. 20 The reactor coolant pump's thrust bearing, high 21 temperature alarm, it was a tailed detector. That is what 22 they found. 23 The check valve, that tour-inch check valve, that 24 took the brunt of the water hammer, its bonnet was lifted, its 25 studs were stretched about halt an inch and then nicely 9
I 48 )
- l 1 hour-glassed and it blew out the gasket.
2 (SLIDE.) 3 MR. MARTIN Check valve-346 is one to the A loop. I 4 The nut was missing. The pin was missing and the disk was 5 just laying in the bottom at the valve. O Check valve-346 was in a similar condition. That is 7 the one to the B loop. 8 Check valve-398 which is in the C loop, its nut was 9 backed oli. It was working its way to the same type at 10 tailure. As a result because it was backed cit, the disk is 11 able to have some treedom so as it approaches its closed 12 position it digs in at the top and that is what happened. It 13 actually dug'in and stuck. 14 CHAIMMAN PALLADINO: Which one is this? 15 WR, MARTIN: This is the one to the C loop which is to fwd-398. 17 CHAIMMAN PALLADINO: You say, " evaluation ongoing." 18 Is there much you can do to evaluate the circumstance with 19 these failures? 20 MM. MANTIN: There may be some additional slide that 21 can be gotten from thermal h}draulic study of what would be 22 the flow conditions in this area and does that establish 23 urnecessary turbulence which with this particular design may 24 cause the disk to spin and back oil, That type of analysis 25 has not been completed.
49 1 COMMISSIONER ASSELSTINE. Are you looking at why 2 some of those parts were missing? That is, what does this say 3 about the maintenance program for the plant as well as the 4 inservice testing program and procedures for the plant? 5 MR. MARTIN: We did go back to the maintenance 6 records. To be quite trank, with the maintenance records we 7 were able to review and we could not reconstruct what 8 maintenance was done. In many cases you could not find _ the 9 records. 10 When you could find the record it would have noise 11 in the B teedwater line check valve. Aha, it sounds like the 12 same thing we had before. We go in it and it says, 13 "r paired." We couldn't find out what actually was done, what 14 repairs were conducted. 15 Until a couple of weeks ago, we didn't know that 16 they had previously had a disk come loose from one of their 17 check valves and they finally found that record. The record 10 system is not that great and what records are available for 19 the maintenance program that has occurred in the past, that is 20 just not a complete record to tell you what happened. 21 In terms of the inservice testing program, the 22 inservice testing program is really not designed to look at 23 this aspect What it does is it tests to see it a check valve 24 will develop ditterential pressure, basically to throttle the 23 tiow. It doesn*t say it is leak tight.
30 1 It each time you test it you are lucky enough that 2 it sinks that time even though it may be wobbling and almost 3 ready to fail, the IST program doesn't check that. It doesn*t 4 open it up and look in which is really one of the ways you
$ could do it.
6 You could also X-ray those check valves and the 7 licensee has done that type of thing when they have known at 8 problems there but this is not part of the IST program. That 9 is not what the IST is really set up for and that was one of 10 our problems when we looked at how NRC had closed the 11 unresolved safety issue. 12 Obviously, they counted on these check valves to 13 work. Where was the regulatory closure? We couldn*t find 14 it. That was one of our problems. 15 COMMISSIONER ASSELSTINE: I remember seeing what you 16 just mentioned about the noise that it occurred earlier in the 17 feedwater line. Do you know enough to say that it they had 18 paid attention to that, that that was a warning sign that 19 would have tipped people off that these things were about 20 ready to come apart? 21 MR. MANTIN: We think so. 22 COMMISSIONEN ASSELSTINE: all right. 23 MR. MARTIN: That is hindsight. 24 COMMISSIONER ASSELSTINE: Yes. 23 MM. MARTIN: When you go into the actual valve, you
51 1 can find little indentations on the downstream side of the 2 valve where this disk was hitting against it after it was 3 loose. That would have been a difficult call The noise 4 didn*t get louder for them. They did do an analysis. They 5 were conscientious enough to try to take X-rays and things of 6 that nature. 7 The noise sounded further downstream because the 8 flow was carrying the noise further, downstream but even in 9 their write-up, they concluded one of the things that could 10 cause this would be a loose disk or a loose nut or things like 11 that. 12 They just didn*l trip to what might be the logical 13 outcome. Their tocus on their analysis was it a part got 14 loose and it went to the steam generator would that cause a 15 problem and they looked at the size of the parts and they 16 said, "That disk is not going to get to the steam generator " 17 COMMISSIONER ASSELSTINE: So they didn*t even look 18 at the fact that " Hey, we really need these check valves. We 19 need to make sure-- 20 MR. MARTIN: They didn*t look at that function of it 21 and quite trankly, none of us on the team would have 22 hypothesized early on that five check valves were going to 23 tail because there are other check valves in a series of 24 these. It la quite an event. 25 COMMISSIONER ASSELSTINE: all right
32 1 MR. MARTIN: The two check valves on the discharge 2 of the feedwater pumps were of similar design and the same 3 type of problems were found. The nuts were loose and they 4 were stuck open. Again, as the nuts get loose, they are able 5 to jam into the seat and they tailed to seat that way. 6 (SLIDE.) 7 MR. MARTIN: A flow control valve which was just 8 upstream on the B side from the check valve, since the check 9 valve didn*t take the water hammer, the next valve down will 10 and the line moved quickly and there is a very tall actuator 11 that goes up and so a lot of inertia and it basically broke 12 one of the arms of the yoke. 13 The B steam generator feedwater line was cracked and 14 it had about an 80-inch crack, in some cases 30 inches in 15 depth. The line was bent. When they tried to unspring it, it 16 wouldn*t settle down without putting a lot of torque on the 17 nozzle, the containment nozzle, and the steam generator nozzle 18 so they finally decided to cut it out and, of course, it is 19 dented where it ran into the concrete wall and some other 20 components. 21 The feedwater line supports and snubbers were 22 damaged. 23 (SLIDE.) 24 MR. MARTIN: The auxiliary feedwater line which 25 comes att of it, comes ott at a 90-degree angle, it obviously
53 1 moved. We can see where the supports were moved but no real 2 damage. The inspections there are pretty good. 3 The containment sphere, early on we were told that 4 there were some crack-like indications about a quarter inch 5 long that you can pick up with very high contrast mag 6 particles. 7 Further investigation at this point has determined 8 that those are not cracks. They were just indications in the 9 surface and they have actually taken them out and everything 10 is fine there. 11 The security access control equipment, there were 12 some problems that caused doors to stay locked. Operators had 13 been prepared for this because of our past bulletins and 14 experience and the operators were very quickly able to cope 15 with this and it was, "I recognize, take care of it and go." 16 There was no safety safeguards problem here. 17 There will be a separate letter by the way to NMSS 18 documenting what we found there because that is safeguards 19 information. 20 (Commission Roberts exits the meeting.) 21 COMMISSIONER ASSELSTINE: On he feedwater system and 22 the water hammer, was the damage about what you expected from 23 the water hammer and the magnitude that occurred here? 24 MR. MARTIN: I am going to have to ask Mr. Serkis to 25 answer that because this is the first big one I have seen.
54 1 COMMISSICNER ASSELSTINE: All right. One thing I 2 was wondering is given'the seismic design of this plant 3 whether that was of benefit or advantage or not 4 MR. SERKIZ: In terms of what we were able to 6 analyze after the event by backing out loads that would be 6 required to stretch the bolts in a particular check valve, the 7 damage that was sustained at the different support junctions 8 as best,we can ascertain the void traction at the time of 9 occurrence was very low, perhaps less than two percent, on the 10 order of one to two percent. 11 The type of calculaticns we have the capability to 12 do today indicate this is the level of damage you might 13 expect. We have not done a real thorough piping analysis to 14 back analyze that this would be the size and shape of the 15 crack. We do know that it extends about 80 to 84 inches, 20 16 to 23 to 30 percent penetration from the outside, and there is 17 still ongoing analysis. 18 COMMISSIONER ASSELSTINE: All right. 19 MR. MARTIN: Next slide, please. 20 (SLIDE.) 21 MR. MARTIN: The event was significant. You had a 22 total loss of AC for about four minutes. You had a total loss 23 of feedwater for about three minutes. You had a severe water 24 hammer that was experienced in the feedwater system that 25 caused a leak, it damaged plant equipment and it challenged
55 1 the integrity of the ultimate heat sink. 2 All'the indicated steam generator water levels did 3 drop below scale and that was largely because the blowdown was 4 left open and the reactor coolant system experienced an 5 acceptable and I stress the acceptable but unnecessary 6 cooldown transient. They were less than the tech spec 100 7 degrees per hour. In fact, I think the peak number was 70 8 degrees per hour. 9 (SLIDE.) 10 MR. MARTIN: Significant findings and these are not 11 necessarily in the order of their significance. The primary 12 cause for the water hammer was obviously the failure of the 13 multiple check valves. Point to be made, all the steam 14 condensation-induced water hammer although it occurred in the 15 B line, conditions were set up for it to occur in the A and C 16 lines, also, it just didn't happen. 17 Even after the leak had developed on the check valve 18 and they kept continuing to feed that line, they were setting 19 themselves them for a second water hammer which didn't occur 20 but the conditions were there. You had steam. You had cold 21 water. You had a pipe that was holding it. 22 COMMISSIONER BEHNTHAL: Why didn't it occur again? 23 MH. MARTIN: It is a catastrophic type thing. You l 24 can meet all of the conditions and it won't occur. 25 (Commissioner Roberts re-enters the meeting.
$6 1 MR. MARTIN: If the water on the top is taking the 2 h e a t- so you develop layers of water and 11 there is not enough 3 turbulence for the cold water to get over and start the 4 catastrophic condensation, the precipitous condensation if you 5 will, that it doesn*t happen.
6 But yuu get a little turbulence or it has to make a 7 turn in a pipe and it comes up to the surface, then you can 8 start it. 9 COMMISSIONER BERNTHAL: It is a little more 10 complicated than a water hammer on your dishwasher at home, I 11 guess, 12 MR. MARTIN: Yes. 13 The tailures of the five check valves in the 14 feedwater system provided the mechanism for a common mode 15 failure of the heat sink. 16 CHAIRMAN PALLADINO: One of these cracks was 20 foot 17 long, was it? 18 MR. MARTIN: That was the rupture in the secondary 19 system in that evaporator. It is about 20-teet long and about 20 two feet wide and it resulted from a single tube rupturing but 21 that was 375 degree water at about 1,300 pounds and then it 22 had a 15-pound shell to get through. 23 CHAIRMAN PALLADINO: There was an 80-inch crack? 24 Was that on the pipe line? 25 MR. MARTIN: That was on the feedwater line in
57 1 containment and it was right near where the pipe experienced
'2 significant westward motion and rammed into the wall, crushed 3 its support there and the crack kind of spirals around the 4 pipe. It is about 80 inches long.
3 The long horizontal runs of feedwater piping with 6 the potential for voiding are particularly susceptible to 7 destructive steam condensation-induced water hammers. The 8 longer they are, you have a bigger space there, more 9 acceleration of the slug and there is more opportunity for the 10 water hammer to occur. 11 Further, operators were not provided the means for 12 detecting the voiding in these lines or given guidance on 13 appropriate ways to deal with the situation. They continued 14 to feed it. They didn't know there was any problem. In fact, 15 they didn't even know the line was voided. This was not 16 information available to them in the control room. Design or 17 procedural changes may be warranted there, one of the basis 18 may be to make the check valves work in the future so you 19 don't get into this condition. 20 (SLIDE.) 21 MR. MARTIN: The flash evaporator failed because it 22 was over pressurized by the discharge of the running pump. 23 CHAIRMAN PALLADINO: Are there no reliet valves in 24 the condensato system? 25 MR. MARTIN: There is one small thermal relief on
dB 1 the third point heater. It is a one-inch relief It is 2 basically just for thermal expansion and contraction in the 3 system. There are reliefs on the steam side of the heaters 4 but they would not have played a role. There was a reliet on 5 the shell of the evaporator that failed. It is a 15-pound 6 relief and we went and checked. It works fine. It just
? wasn*t able to handle the magnitude or the amount of steam and 8 water that was injected into it.
9 (Commissioner Bernthal exits the meeting.) 10 CHAIRMAN PALLADINO: There aren*t reqairements for 11 how much it must handle? 12 MR. MARTIN: You are getting me outside my area of 13 expertise. I will have to ask Vic on that one. 14 MR. STELLO: I believe ASME code has some conditions 15 you have to meet fur those portions for which the code 16 applies. I don't know how far the code went back. 17 CHAIMMAN PALLADINO: Maybe that is something we can 18 ask Southern Cal. 19 COMMISSIONER ASSELSTINE: Are there generic aspects 20 to the feedwater problems that occurred here or are they all a 21 tunction of the unique design aspects for this particular 22 plant? 23 MR. MARTIN: We did not go and look at other plants 24 so I really can't answer that question. 25 COMMISSIONER ASSELSTINE: Does it raise questions in
59 1 your mind though about whether we ought to look at these long 2 runs, whetner we ought to look at check valves, those kinds of 3 things to see 11 there are potential problems at other plants? 4 MR. MARTIN: It does raise that question. 5 COMMISSIONER ASSELSTINE: All right. 6 CHAIRMAN PALLADINO: Especially check valves for 7 this design. B COMMISSIONER ASSELSTINE: Yes. 9 MR. MARTIN: The timing of the five check valve 10 tailures could not ascertained with certainty. The team 11 concluded that all check valves had failed prior to the event 12 because the missing parts to the valves were'not found in the 13 inspected piping which was going in the upstream direction. 14 We have not looked at the steam generator yet so the licensee 15 is only starting into his inspection of those. 16 CHAIRMAN PALLADINO: There is no way of knowing when 17 a check valve has tailed? 18 MR. MARTIN: There are a couple of ways. If you 19 suspect it, then you c ;.n either dissemble it and look at it 20 that way or if you have rather stable conditions, you might 21 radiograph it and see that it is up and out of the flow 22 stream. 23 Check valves are supposed to be designed so that at 24 tull flow, they are up and out at the flow stream so they 25 are not subject to the turbulence. Further, they are supposed
1
)
60 i l 1 to be far enough downstream from obstructions and things that 2 produce turbulence that are not subject to these kinds of -- 3 CHAIRMAN PALLADINO: Except in part 1 cads or part 4 flows.
$ MR. MARTIN: Exactly. You don't normally operate at 6 part loads if you are capable.
? CHAIRMAN PALLADINO: Aren't there hydrostatic tests 8 on these systems and wouldn't you be able to tell between 9 certain parts of the system that the check valve is either 10 closed?
11 MR. MARTIN: I think the next one answers that. 12 CHAIRMAN PALLADINO: All right. 13 (Commissioner Bernthal re-enters the meeting.) 14 MR. MARTIN: There is a surveillance procedure for 15 testing these check valves. It is called the inservice 16 testing program. The program is designed to require you to 17 perform tests to make sure valves can perform their safety 18 function. These check valves, their function was to close to 19 prevent the auxiliary feedwater from coming back from the 20 steam generators. 21 The test that they had designed was that with the 22 plant down and relatively ecol, the water in the steam 23 generator which is only about ten feet higher would provide 24 about a five pound head of water which would hold the check 25 valves shut.
)
61 1 They would then isolate upstream open a drain and 11 2 the water coming out of the drain slowed down and stopped or 3 was minimal, they would say, "Aha, the check valve is 4 holding." But you have this one-inch line. You have the 5 tubing running across and the call was very subjective, 6 minimal flow. 7 Is that minimal flow because you have a small head 8 of water up there, only five pounds across this valve, or is 9 it because the check valve is really doing its job? It is a 10 ditticult call. They recognize that it is a difficult call 11 In fact, they performed additional tests at mode three where 12 they are up at about 700 pounds. 11 you have a hose, you can 13 tell whether it is 700 pounds coming. The check valve either 14 is holding or not at that point. 15 That was one of the tests that they did for the 16 discharge of the feed stations. Down at the feedpumps what 17 they did was they shut the isolation valve on the check valve 18 to be tested, they turned on the pump on the opposite side, 19 made sure that it was feeding the steam generator showin2 that 20 they had a flow path. Then they opened the isolation valve 21 and looked at the discharge pressure from the idle pump. Did 22 that discharge pressure go up? 23 But they didn't check the calibration of the gauge 24 beforehand and, in fact, there is an open path from the 25 suction of that idle pump all the way back to the other one.
i 62
' l 1 So unless you have significant leakage through 2 there, you are not going to be able to tell whether that 3 check valve was really shut.
4 COMMISSIONEM ASSELSTINE: We spent a lot of time 5 yesterday talking about surveillance testing and the need to 6 eliminate unnecessary surveillance testing. It sounds like 7 this tells us that in some instances what we may need to do is 8 look at those tests and make sure that they really are 9 effective in identifying the condition of components in the 10 plant. 11 MR. MARTIN: I agree. 12 (SLIDE: ) 13 MR. MARTIN: The NRC has not completed its review of 14 the SCE's inservice testing program and there have discussions 15 back and forth between the licensee. 16 COMMISSIONER ASSELSTINE: That one sounds a lot like 17 the auxiliary feedwater pump question at Davis Bessie. What 18 has been the reason why the statt and the licensee have been 19 arguing back and forth since 1977? 20 MR. MARTIN: Part of it is that it is an old plant 21 that wasn't designed for this testing. Where the testing was 22 easy to conform with the regulations, the licensee did. In 23 discussing with the NRC people and in discussing with the 24 licensee and looking at the documents that went back and 25 forth, there was concern on the NHC side that the licensee was
I e3 1 i 1 not aggressively trying to pursue resolution of these things, i 2 In fact, there are even documents to that effect in ! 3 the file. At the same time, NRC wasn't very aggressive in 4 trying to do the review. They had other reviews to do and so 5 the project manager was pushing to get some of the interviews 6 done on time. There is blame on both sides. 7 COMMISSIONER ASSELSTINE: What arguments was SCE 8 using for not having to do these tests? 9 MR. MARTIN: There were problems where the design 10 did not make testing very feasible, would have not 11 conclusively given information on whether the valves were 12 performing in the way they were. They were not only on these 13 check valves. The whole program looks at lots of valves and 14 there were specific disagreements and open issues that stayed 15 open for very long periods of time. 16 COMMISSIONER ASSELSTINE: Are the documents that you 17 describe going back and forth, are those in the report? i 18 MR. MARTIN: They are not in the report but they are 19 going into the PDH. 20 COltMI S S I ONEli ASSELSTINE: All right. Good. 21 COMMISSIONER BERNTHAL: When I was out there a year 22 and halt or so ago prior to their start-up, I recall and 23 haven't forgotten in the discussions that we had on tech specs 24 and the slowness with which we seem to be able to respond to 25 the proposed tech spec changes that this utility for this i
)
64 1 plant had something like 70 pending tech spec changes. Maybe 2 I should go beyond that. I believe that broadly speaking 3 these were license amendments that they were proposing. It 4 wasn't just tech specs. 5 Was there any relationship between this large 6 backlog of license amendments? I don't know whether that has 7 been cleared to this day. It certainly was not in prospect 8 that those would be rapidly disposed of when I was there. Did 9 you find any relationship between that big backlog of work? 10 MR. MARTIN: We did not look at that. 11 COMMISSIONER BERNTHAL: You don't know. 12 MR. MARTIN: 1 just don't know. 13 COMMISSIONER ASSELSTINE: That is a good question 14 though, Fred. That's right. My recollection is the same as 15 yours. There was a large backlog of license amendment 16 requests in. I think a lot of those were more on procedural 17 matters, changing the organization, names and boxes on the 18 organization chart and things like that. But it would be 19 interesting to know it any of those affected this testing 20 program. 21 COMMISSIONEH BENNTHAL: Whether or not they attected 22 hardaare, I tnink there may very well be a tendency to freeze 23 the mindset of the people involved here while they are waiting 24 for us to respond to what they clearly viewed at that time as 25 an unacceptable backlog of license amendments that they didn't i i i 1 1
65 1 seem to be able to get changed in a timely manner, that they 2 thought were important to the operation of the plant. , 1 3 I am still confused. Vic, maybe this is something 4 that we need to get resolved once and for all. I am still , 5 confused by the staff position which seems to oscillate 6 between'Sholly is a problem and Sholly isn't a problem on thia 7 matter of license amendments. 8 I seem to be getting different signals almost every 9 time we meet on that issue. 10 MR. STELLO: If the number is anywhere near 70, it 11 is not a Sholly issue. It has to be workload and commitment 12 to get it done. 13 COMMISSIONER ASSELSTINE: That's right. 14 MR. STELLO: It is a good question and you deserve 1$ an answer and you will have one. 16 COMMISSIONER BERNTHAL: Not specifically for this 17 plant but I recall very well when we were talking about 18 license amendments and the big backlog, Sho11y was cited as a 19 big time waster, a big problem for the staff and then a month 20 or two later I think in response to a written inquiry that I 21 made the stati came back and said, "No, this wasn't a 22 problem." 23 I am not sure where we stand on that to this day. I 24 am confused. 25 , CHAIRMAN PALLADINO: Let's try to get an answer. 4
66 1 MM. STELLO: Let me sharpen up my answer. You have 2 raised a good question with respect to why is there the 3 backlog and you deserve an answer but maybe even far more 4 important, was that backlog in any way shape or form -- 5 COMMISSIONER BERNTHAL: For this case. 6 MR. STELLO: -- in this case caused a problem which
? could have been avoided had we dealt with the issue. That, to 8 me, is an even more important question and we will get that 9 answer, too.
10 COMMISSIONER ASSELSTINE: Good. 11 MR. MARTIN: Because the water hammer occurred and 12 we knew that the water hammer unresolved safety issue had been 13 resolved, we went into what happened here and we found that 14 the USI did not specifically address the prevention and 15 mitigation of the consequences of r condensation-induced water 16 hammer in the feedwater piping. 17 We interviewed N1? C statt to find out was that an 18 oversight or why did you do it and we didn't really find a 19 good answer there. 20 COMMISSIONER ASSELSTINE: Was it considered? 21 MR. MARTIN: There was evidence that check valve 22 problems were known and in some of the NUMEG's that say check 23 valves can get you into trouble but it kind of dropped there. 24 We went into looking at the regulatory analysis of l 25 the USI that is another NUREG and there the statt acknowledges
.' . l 67 1 that the elimination of water hammer is not feasible, that we 2 have steam, we have water in these plants and it is going to 3 happen some day. 4 So you can't completely eliminate it. They did note 5 though that all of the fixes that had been done in the late 6 1970's, that the frequency of water hammers had been 7 substantially reduced and this really was the result of 8 design, operations, training people, sensitizing people that 9 what conditions are necessary and how to avoid them. 10 Finally, the studies showed that the water hammers 11 that we had previously been very concerned about were really a 12 less significant satoty concern than we had previously 13 hypothesized. 14 We had no example of where we had had a release of 15 radioactivity. We had had no complete loss of a safety 16 system. Those were the kinds of things that they were relying 17 upon. 18 Based upon our discussions with the statt it looked 19 like we focused on things where we had experience of water 20 hammers and very few reports of water hammers associated with 21 check valves had really come in. There was one from some 22 plant in Europe but that was the only one at the time that the 23 statt was aware of 24 So it appeared tc us that further consideration of 25 the water hammers due to main feedwater line voiding was not
h 68 1 pursued due to a lack of reported occurrences in U.S. plants. 2 That is a conclusion. We didn't really find somebody who 3 said, "That is the reason." 4 CHAIRMAN PALLADINO: What does this say about the 5 resolution of the water hammer? 6 MR. MARTIN: It basically says that it is based upon
? the significance of the thing and the safety of water hammers 8 and they had forced the frequency of water hammers to a very 9 low frequency. They had resolved all of the significant ones 10 that were causing problems, the major contributors to water 11 hammers.
12 They had t r a i s.e d operators on how to deal with water 13 hammers and how to prevent setting up conditions. 14 CHAIRMAN PALLADINO: Have they trained on this 15 particular type of water hammer? 16 MR. MANTIN: No. 17 CHAIRMAN PALLADINO: Should we go back and 18 re-examine this USI? That is what i am basically asking. 19 Maybe that is just a question to t,51nk about. 20 COMMISSIONER ASSELSTINE: From the standpoint of 21 this initiator to water hammers, it sounds like what you are 1 22 saying is that people understood that this was a potential 1 23 water hammer problem. It was dropped. It wasn't factored 24 into the resolution of the USI and you cra't a find a specific 25 reason why it was dropped but the best conclusion you have
69 1 1 been able to come up with is because one had not occurred in 2 the U.S., people just didn't feel that it was worth pursuing. 3 We would wait for a problem to occur. 4 MM. MARTIN: All of the fixes were directed against 5 the major contributors to water hammer. 6 CHAIRMAN PALLADINO: May I suggest that the statt 7 ought to re-examine to see whether anything else should be 8 done on this USI. 9 MR. STELLO: I think clearly, yes, we have to but I 10 think we need to start somewhere and I think Tim is going to 11 be talking more about the check valves and I think we need to 12 devote a great deal of attention to the check valves and it we 13 could assure ourselves that we are getting performance out of 14 the check valves, then we will feel lesser in terms of the 15 need to get into a lot of water hammer issues. 16 Check valves I think are a far more significant 17 issue since they not only contribute to water hammer but they 18 can create a lot more problems. 19 CHAIRMAN PALLADINO: Incidentally, since you went 20 back on check valves, my technical assistant reminds me that 21 there is a check valve in the low pressure RHH system that in 20 case of failure could by-pass containment in this E type of 23 configuration. 24 Are these the same kind of valves? If you can't 25 check these check valves, how can you check that check valve?
70 1 It just expands the question and maybe something else ought to 1 2 be looked at. l 3 MR. MARTIN: That particular check valve you point 4 to is not of the same design. 5 CHAIRMAN PALLADINO: Not of the same design. 6 MR. MARTIN: There were 13 of this design in the 7 plant. All but one have been checked and that one is 8 scheduled to be checked. 9 CHAIRMAN PALLADINO: I see. Is there a way of 10 testing its integrity as part of the surveillance program? 11 MR. STELLO: With respect to those valves, the event 12 B issue that arose out of WASH-1400, my recollection is that 13 we did issue a letter requiring a series of modifications and 14 testing for check valves in all the plants. I am sure that 15 information is available. Let us dig it out and we will get 16 something back to you. 17 COMMISSIONER ASSELSTINE. That could potentially 18 include check valves in these other systems as well or just 19 the ones on event B as potentials? 20 MR. STELLO: I don't believe it included these as i 21 recall 22 COMMISSIONER ASSELSTINE: All right. 23 MR. STELLO: I think this creates at least in my 24 mind a series at different questions and issues that we have 25 to deal with. It doesn*t tall within the same family as the
71
~
1 issue we dealt with five, six, seven or eight years ago. 2 COMMISSIONER ASSELSTINE: all right. 3 COMMISSIONER BERNTHAL: What level of redundancy is 4 there though, how many levels of defense if you will, are
$ there related to these check valves against the water hammer 6 type event? In other words, would the failure or did the 7 tailure of a single check valve, was that sufficient condition 8 to lead to the water hammer?
9 I don't remember all your diagrams. How many of 10 those things had to fail for the water hammer to occur? 11 MR. MARTIN: You needed at least two. 12 COMMISSIONER BERNTHAL: At least two? 13 MR. MARTIN: That's right. 14 COMMISSIONER ASSELSTINE: You had five fail, right. 15 MR. MARTIN: You had five, yes. Three up front are 16 in series with the two behind. You needed one in each group 17 to tail to provide the vent path. 18 COMMISSIONER BERNTHAL: It it were only one, I would 19 be inclined to think that that is not very much defense 20 against a serious water hammer event Two is better. 21 COMMISSIONER ASSELSTINE: But that assumes that a ! 22 single tailure is a pretty high likelihood, that that is all I 23 you are going to have. 24 COMMISSIONER BERNTHAL: Yes. 25 COMMISSIONER ASSELSTINE: And here we had multiple 1
o' . 72 1 failures. 2 (SLIDE.) 3 MR. MARTIN: Going on with the valve i s s.u e , we 4 looked at what NRC relied upon ;or the steam generator water 5 hammer which was a major part in the resolution of the USI. 6 The "J" tubes basically sit on top of the feedring and it was
? designed to prevent draining of those feedrings.
8 That obviously puts a lot of faith on the check 9 valves back in the back that are not draining. So we looked 10 to see 11 there was any closure on this, did they pass it off 11 to the IST people? Did they tell the IST people this is an 12 important thing? We are counting on you to make sure that 13 these have integrity. Quite trankly, we could find no such 14 closure. 15 COMMISSIONER ASSELSTINE: It sounds like we relied 16 on a high level of performance by these valves without taking 17 the next step to make sure there is a program in place that is 18 going to assure that high level of performance. 19 MR. MARTIN: We found no such closure. 20 COMMISSIONER ASSELSTINE: all right. 21 MR. MARTIN: The root cause of the loss of power was 22 a phase-to-phase fault. We talked a little bit about the 23 design of the plant and some of the problems with the diesels 24 and that is really the next finding. 25 (SLIDE.) i 1 j
I
?3 l 1 MR. MARTIN: We talked a little bit about the loss 2 of voltage auto transfer scheme and why it didn't work.
3 We were particularly concerned about the multiple 4 spurious indications early in the event of the safety 5 injection actuation. We felt that this added to the 6 operator's burden. That they were able to diagnose it and 7 take the right action is to their credit. But we felt that 8 the design certainly of the annunciator, safety injection 9 actuation when you have a loss of power is inappropriate. It 10 should not come in. That is a design deficiency. What caused 11 the other problem is still unknown. 12 We pointed out a problem with the operating staff, 13 with the concurrence of management, followed procedures which i 14 we feel were inappropriate for the troubleshooting in the 15 electrical ground and we talked a little bit about that. 16 Once the electrical ground was located instead of 17 promptly de-energizing that transformer, they again went and 10 tried to find other ways to avoid shutting that thing down 19 promptly, seeing 11 they could find something they could take 20 off the bus that would not lead to the shutdown. 21 The inspections they performed added a little bit of 22 danger to the personnel because now you go out on grounded 23 equipment. Equipment is not suppose to have any voltage on 24 it. The casings and things like that may have some voltage on 25 it and we didn't feel that that was a good idea.
74 1 CHAIRMAN PALLADINO: For planning purposes, do you 2 have an estimate on how much longer you will be? 3 MR. MARTIN: I can move through here fairly quickly. 4 CHAIMMAN PALLADINO: 1 don't want to rush you. 5 MR. MARTIN: t?ach one basically points out our o findings. They are in our report. Let me jump forward -- 7 CHAIRMAN PALLADINO: I do want to talk a little bit 8 about follow-on actions. 9 MM. MARTIN: Let me jump all the way to what I 10 believe is the most significant aspect of this and I think 11 that is really important Let's go to slide 33. 12 (SLIDE.) 13 MR. MARTIN: Our feeling was that the most 14 significant aspect et this event was the failure of five 15 safety related feedwater check valves, that they had degraded 16 to the point of knoperability and in a period of less than a 17 year after we knew tests had been performed that we had le confidence in even though some of the tests were not described 19 in procedures. The tests that were ultimately performed we 20 telt we could be confident that the valve functioned that 21 time. 22 The root cause of the check valve tailures has yet 23 to be determined but potential contributors might be 24 inadequate maintenance, inadequate inservice testing, 25 inadequate design and inadequate consideration of the ettects
?$
1 of reduced power operations. 2 COMMISSIONER ZECH: I agree with that conclusion. 3 That is a very important conclusion and I think it is 4 something that really needs pursuing because it looks to me 5 like it could be involved with other things, water hammer and 6 so forth. I think you are right on with your conclusion.
? COMMISSIONER ASSELSTINE: What was the last rating 8 of this licensee in a SALP evaluation for maintenance and 9 surveillance testing?
10 MR. MARTIN: I don't remember. 11 COMMISSIONER ASSELSTINE: It would be interesting to 12 know that. 13 MR. MARTIN: I don't have it with me. 14 COMMISSIONER BERNTHAL; The one thing that you don *t 15 say directly here in your final conclusion, you don't address 16 the question of whether this plant's long outage and for want 17 of a better term how they mothballed the system during that 18 long outage might have afie;ted some of the components. 19 In other words, that is unusual. You don't have 20 normal operation and we hear all sorts of things about 21 bacteria-induced corroston and Lord knows what these days that 22 we didn't used to know about. Did you think about that at 23 all, whether that long outage perhaps inappropriate 24 mothballing techniques might have contributed to this multiple 25 deterioration?
I i 76 ; 1 MR. MALIT IN : We certainly thought about it but when 2 we looked at the actual damage it looked like the check valves 3 were boing subject to impact loads and that you are not going 4 to get in a shutdown condition. 5 In some cases the stud was mushroomed over, it had 6 been beat so many times. In other cases, it looked like even 7 11 there had been a nut on there, it would have been shattered 8 by these impacts. Others, the nut was just backed ott, the 9 threads were in fairly good shape. 10 I don't know that that is not a problem but it 11 didn't look like -- it looked like a service related problem 12 and not the mothballing problem. 13 COMMISSIONER ASSELSTINE: Did you say though that 14 there were tests that were done before the plant started up 15 that led to the conclusion that in fact they were in pretty 16 good shape at that point and they degraded substantially 17 during this year of operation? t8 NR. MARTIN: There were tests since then in October 19 and November of 1984, in February of 1985 and then the event 20 occurred in November. 21 COMMISSIONEH ASSELSTINE: All right. 22 MR. MARTIN: That is my presentation. 23 CHAIRMAN PALLADINO: Thank you. I had one question 24 I wanted to ask. In the report in one of the sections you 25 talk about the inquiring attitude of the people at San Onotre
??
1 Unit-1 and you comment that they are not inquiring enough. In 2 view of that, how can we have assurance that the follow-up on 3 San Oncire Unit-1 is going to be realistic and sufficient? 4 MR. MARTIN: Let me answer that. Once we identified 5 the quarantine list we have worked wi'.h the licensee to 6 devetop an action plan to approach each one of those and to
/ resolve what was the root cause of these problems.
B CHAIRMAN PALLADINO: Uut at several times you 9 identified situations that they had not even asked questions 10 about so it looks like there may be dependence on NRC finding 11 everything whereas there ought to be certainly some effort on 12 San Onotre*s part. 13 MR. MARTIN: We have an unfortunate advantage on 14 them. We are used to investigating. I have been in multiple 15 investigations and I have also had some training on how to 16 investigate. We don *t accept the easy answer. It it doesn*t
!? hold water across the board, every chart, et cetera, we don't 16 let it tall.
i 19 We were pressing the licensee to find a lot of these 20 things. I am not sure that maybe down the road they wouldn't 21 have found them but when they would respond to our concerns 22 with the easy answer and we said that that doesn't cut it for 23 this, this and this reason, the licensee after he saw the 24 logic of this rationale would go in and do the investigations. 25 Quite trankly, their disassembly and inspections 1
I 1 78 1 were excellent. There were a few examples like that where it 1 2 left us with the concern that had we not kicked off the 3 investigation some of these things might have been missed and 4 that is a human-driven thing. It is part of how people are 5 trained to do investigations, how inquiring they are. 6 CHAIRMAN PALLADINO: But still that doesn't leave 7 you with a good sense of confidence and I don't know if there 8 is anything that you might have found that could help. Jim, 9 go ahead. 10 COMMISSIONEN ASSELSTINE: Joe, I was just going to 11 suggest that it seems to me I think your question is exactly
, 12 the right one. 1 guess the question I have is, is the staff 13 going to insist upon the same kind of responsive program by 14 the licensee that we have seen from Toledo Edison on Davis 15 Bessie?
16 That is a rigorous search for the root causes on 17 many of these areas where you have said so far at least based 18 upon the investigative team's review, you haven't been able to 19 find those root causes. review of the systems that were 20 involved here to identity what the problems were and then a 21 corrective program to make sure that those things get 22 corrected together with a people program to make sure that 23 that kind of inquisitiveness that seems to have been missing 24 wasn't there, tix the training problems with these folks. 25 In essence, don't we need a corrective program for 1 i l
79 1 the licensee that is their response to the kinds of problems 2 the team has been able to identify? 3 COMMISSIONER BERNTHAL: I can't help but observe 4 that inquisitiveness is something that is great in almost 5 ali circumstances perhaps except in emergencies during an 6 accident. Commissioner Zech observed earlier and I think 7 correctly or wondered at least out loud why certain procedures 8 were not followed and there is just an inherent conflict 9 there. 10 I don't think we should lose sight of the positive 11 things that you have said about the performance of these 12 operators but inquisitiveness in emergencies under duress is a 13 trade off. 11 you are too inquisitive you are inclined to 14 say, "Well, I wonder 11 that is really the right procedure and 15 maybe I ought to be creative here." 16 We have learned over and over that that is not a 17 very wise thing to try to do. 18 CHATRHAN PALLADINO: I wasn't talking about the 19 inquisitive during the accident but rather now in looking at 20 the plant and seeing what needs to be corrected so that the 21 plant can operate safely. 22 I have one other question. Is the understanding 23 between the staff and NRC such that San Onofre won *t start up 24 without NRC approval? 25 MR. STELLO: Yes. Let me try to respond. You are
80 1 asking good questions. They are proper questions. We need to 2 have answers. We need to be sure that the licensee is going 3 to take what is in this report seriously and aggressively seek 4 to find those root causes. We have to look at not just what 5 we have learned on this plant and as applied to this plant but 6 we have to aggressively look at what this means to the
? industry, the generic implications and assure that there are 8 programs for those.
9 We need to do all of those things. We haven't done 10 them yet though. So we are not prepared to talk about them 11 and we need to have time to do them so that we can do them 12 well 13 The industry has to look at these lessons and they 14 need to respond, not just San Onotre but industry at large. 15 We have had pieces of paper to go out to alert the 16 industry of these kinds of things and we will be looking to 17 see how they are reacting to them. You simply cannot do that 18 without the time. 19 We have only had this report for two days and that 20 simply is not enough time. 21 CHAIMMAN PALLADINO: I am glad to hear you say that 22 because the impression one might get from reading the report 23 is that everything is done. 24 MM. STELLO: It is just beginning. 25 COMMISSIONER ASSELSTINE: You are just getting
gi 1 started. 2 CHAIRMAN PALLADINO: All right. What I am getting 3 at is I would like to see the s t.a t t develop a tollow-on plan 4 and I think San Onotre, Southern Cal has to develop its 5 follow-on plan and we ought to see how the two mesh where they 6 are suppose to mesh. 7 MR. STELLO: I will add a third element, a need for 8 the industry at large for them to look at it. 9 CHAIRMAN PALLADINO: Yes. 10 MM. STELLO: I want to feed this kind of information 11 into our inspection programs and make sure that we do a better 12 Job looking at it I want to make sure that INPO becomes 13 sensitive to it as they do their job and they can put all this 14 in. 16 In that context I am almost getting into Rancho Seco 16 but just let me peep in for a moment. We have asked INFO to 17 identity an individual who can start to participate in some of 18 these teams because we are convinced the kind of information 19 developed in here is going to be very, very helpful and useful l 20 to them and for them to have first hand experience with what 21 we are doing and in Rancho Seco we have the first step in that 22 process to have them come in and participate. 23 We will talk about that in a minute. But, yes, we 24 are going to do all these things and what we need is the time 25 to do them and do them well. I don *t think we want to do them
82 ,- 1 part way. We want to do a good job. 2 CHAIRMAN PALLADINO: That is all I need for the 3 moment. Let me just touch on one other thing and I am sure i 4 there are a lot of other questions. I did want to suggest 5 that we not only give two minutes today for San Onotre people 6 to make a comment but that we actually arrange a meeting which
? we can do in terms of agenda planning.
8 COMMISSIONER ASSELSTINE: good. 9 COMMISSIONER ZECH; I agree. 10 COMMISSIONER ASSELSTINE: Give them a few weeks and 11 let them come back and tell us what they propose to do. 12 CHAIRMAN PALLADINO: Let me see 11 there are any 13 further questions. 14 COMMISSIONER ROBERTS: Would you comment on the 15 efficacy of the IIT concept? 16 MR. STELLO: This is the second one. 17 COMMISSIONER ROBERTS: Hight. 18 MR. STELLO: We have one more. I think there is an 19 awful lot of learning to do about how to just go about the 20 business of it. I chatted with Jack. I think we have done 21 not early enough in alerting the industry, in having some 22 seminars out there, telling the industry how we are going 23 about this activity. 24 I sense that the whole process is very, very 25 disruptive to utilities. It needs to be done and I think we
83 1 are learning a great deal from them but I think I would like 2 more time to look at what we are getting out them. It is 3 clearly a resource issue. We want to make sure that we are 4 getting what we want out of the resources that we are 5 applying. 6 Based on the second one, I think they have done an 7 excellent job in looking at it. It is a pretty complete job. 8 How we continue to do them in the future I want to open it up 9 and allow for opportunity for comment, get some interchange 10 going with the industry to make it better. We haven't done 11 that yet and we need to do it. We have work to do on the 12 process itselt 13 But I think based on the time we have had to develop 14 it in pretty good shape. 15 COMMISSIONER ZECH: Just a brief comment if I may. 16 First of all, I think certainly this is a very significant 17 event as you have pointed out and requires follow through on 18 the part of the stati and the Commission. There is no 19 question about that. 20 The only other thing that I would like to say is I 21 do think that this team has really done an excellent job of 22 investigating and reporting. I think your report although we 23 haven't had it but a day or so. I think it is an excellent 24 product and I really think that the whole endeavor as 25 disruptive as it may be and I agree that it probably is, but
l 1 l 84 1 it is a step in the right direction, to be more responsible, 2 to involve ourselves in what I think is a constructive way to 3 increase plant safety and reliability. 4 I think this team has done an excellent job and I 5 commend them for their efforts. 6 CHAIRMAN PALLADINO: I thank the Commission support 7 staff, too. 8 COMMISSIONER BERNTHAL: I have a couple of questions 9 that I wanted to -- well, I guess one of them is really a 10 comment. 11 We keep coming back to this loss of power. This 12 wasn't a station blackout issue. Almost on a few weekly 13 basis we see loss of power, near station blackout or at least 14 the clear potential of a station blackout. 15 I would just suggest that again this points to the 16 importance of this station blackout issue in the rule that the 17 Commission has before it on the station blackout matter and 18 again the fact that the measures we proposed are rather modest 19 compared to the measures that all of the European leaders have 20 taken in this particular area. I think we ought to get moving 21 on it. 22 The other point I wanted to inquire about was let 23 me read you at least the date and one key phrase from one of 24 the NRC's press releases of last summer. We were speaking 25 earlier about having looked at this plant rather carefully.
85 1 This was one of the so-called older plants that we 2 evaluated under our systematic evaluation program and last 3 June 12th we issued a press release that among other things 4 and it at some length went on, but it among other things said 5 that significant areas of safety examined in systematic 6 reviews include systems required for safe shutdown of a plant 7 and lots of other things. 8 It leads one to ask how effective that program was 9 and how carefully the plant was looked at in that program. 10 How well did the systematic evaluation work? Was this being 11 pre-ISAP, the Integrated Safety Assessment Program for the 12 benefit of the press over there and alphabet soup, since it 13 pre-dated, I gather, and is somewhat different from ISAP, 14 would we do better today or shouldn't we have expected to pick i 15 up some of these problems when we carried out this so-called 16 systematic evaluation? 17 MR. STELLO: I think it would be unfair to criticize 18 the thoroughness of the problems + hat we looked at in the past 19 and how well we did our job. 20 We have a number of generic issues that fall within 21 the gamut of the . ad of experience you are seeing here, a 22 variety of system interactions kinds of problems, that are 23 very, very difficult to analy:e. They are not easy. 24 You are going to pick up some of these things in 25 spite of however much effort you put into them by experience. l
86 1 You are going to continue to see events. You can't preclude 2 an event from happening in a plant, 3 I don't think we have the tools to do that kind of a 4 review to the degree of perfection you would need to. 5 ~ COMMISSIONER BERNTHAL: There are 21 failures. I 6 appreciate what you are saying, Vic, and I am not criticizing 7 statt as individuals. I am maybe asking how ettective some of 8 these programs are in design rather than asking whether 9 somebody really did his job. 10 There are 21 tailures here. It may be that we were 11 skewed. As was pointed out earlier, it may be that we were 12 skewed too much toward the seismic issue down there and that 13 some of the other analysis was slighted. I don't know. But I 14 am asking about the program design perhaps. 14 MR. STELLO: Let me step back a little bit. To 16 contrast what really happened here and the kinds of review, 17 it.you look at it you had two check valves at least two in 18 series that had to tail coincident with feedwater being 19 disrupted so that the lines could empty out to get the kind of 20 water hammer that you did even though you had as Tim correctly 21 pointed out that condition set out into the other lines. You 22 didn't get it but you did in the one. 23 You are talking about really three layers of "what 24 it " If you look at all of the combinations that you can have 25 in those plants with being able to pick up the interactions
1 87 1 you can develop, that is a vary, very ditticult and broad 2 issue to deal with. I think it comes within the gamut of the 3 kinds of systems interactions that you can have in plants. 4 COMMISSIONER BERNTHAL: We did a PRA on this plant, 5 'did we not't Wouldn't you have expected that to pick more of 6 these things u p '! 7 MR. STELLO: PRA by itself is a very important tool 8 to use in safety assessment but I don't believe that one can
'9 be confident that PHA is going to pick up systems interaction 10 kind of problems. It doesn't get into those kinds of things. '
11 They are usolul and helpful and give you insights but I think 12 the experience we have is that they are not a substitute for 13 looking at this broader issue which we are looking at. 14 Until it is finished and we start getting some of 15 those kinds of insights as to how to do reviews for these 16 kinds of problems, then we won't have complete answers. 17 COMMISSIONER ASSELSTINE: It goes back to 18 maintenance. 19 COMMISSIONER BERNTHAL: Maintenance, of course, is 20 assumed throughout all of that. It sounds like you are saying 21 that it was unreasonable to expect that that program even 22 given PRA would have been able to pick up this sort of event. 23 MR. STELLO: 1 could understand how it could not and 24 there is this block that we are still looking at called 25 systems interaction which we need to finish as an issue to
88 1 help us get some more tools and insight of hcw to do this 2 better. 3 We are working in the area but until we get that 4 kind of information to help us, we won't be able to do our 5 jobs as well as we would like to. 6 CHAIRMAN PALLADINO; I think you do raise a very 7 important point. Do you have a final comment? 8 COMMISSIONER ASSELSTINE: Just a couple of 9 comments. I think, Fred, you have hit on one of the right 10 questions and Vic, you are probably right. You are not going 11 to be able to eliminate all operating events but in essence, 12 the need for 11T's is going to depend upon how well we 13 identity and anticipate some of the vulnerabilities that led 14 to this kind of event as well as the ones that led to Davis 15 Bessie, Mancho Seco and the others. 16 I think Fred is right, that it is worth taking a 17 critical look not from the standpoint of did someone not do 18 their job but more from the standpoint of when we look at the 19 root causes of this and these other events, what do we need to 20 do better to try and identity what may be design 21 vulnerabilities, what may well be operational deficiencies and 22 training deficiencies and what may well be maintenance and 23 testing deficiencies in this case to make sure that we can 24 minimize at least the chance of seeing more of these things in 25 the future.
, 89 1 That really ought to be part of the look. I think 2 while the IIT did a superb Job and I agree with the comments 3 that Lando made earlier, looking at the facts and finding out 4 what happened here, one element that I think is still missing 5 is that critical self evaluation. What does this tell us 6 about weaknesses both in the licensee's programs and in our 7 own that enable these kinds of events to occur and what can we 8 do to help correct those problems for the future?
9 CHAIRMAN PALLADINO: We put together an advisory 10 group on Davis Hessie. 11 COMMISSIONER ASSELSTINE: It is Just going to get 12 started on that issue. That*s right. 13 CHAIRMAN PALLADINO: Yes, Maybe we want to think 14 further about whether the charter of that group should be 15 expanded but let's not take that up now. 16 COMMISSIONER BERNTHAL: Why don't we Just make it a 17 permanent group. 1 18 LLaughter.) 19 CHAIRMAN PALLADINO: We may find that that is 20 necessary. 21 COMMISSIONER ASSELSTINE: It sounds good to me. 22 COMMISSIONER BEINTHAL: I couldn't resist. 23 (Laughter.) 24 CHAIRMAN PALLADANO: Incidentally, 1 think it is 25 worth looking at.
90 1 Let me suggest that we give San Onotre 2 representatives a couple of minutes since they requested it 3 and then could I get an estimate.of how much time you think 4 you need on Hancho Seco? 5 MR. STELLO: It we could get five minutes 11 6 everyone would-- 7 CHAIRMAN PALLADINO: Restrain ourselves. I am 8 willing to give tive but if the Commission is willing only to 9 give two, we will give two. 10 Let me ask the representatives from San Onofre to 11 make their comments. 12 MR. PAPAY: Thank you, Mr. Chairman. My name is 13 Lawrence Papay. I am the senior vice president of the 14 Southern California Edison Company. I am joined today by Ken 15 Baskin who is our vice president of nuclear engineering safety 16 and licensing. 17 1 know you said two minutes. I would like to 18 stretch that if I might to a little bit more because of the 19 significance of the investigation and the significance of l 20 some of the consequences. 21 Quite frankly, we share with you the fact that this 22 is a significant event. We have nad analyses under way and we 23 continue to have analyses under way and are looking at root ! 24 causes and all the other items that are addressed. 25 I would like to second your statement concerning the
91
~
1 desirability of a tollow-up meeting after we have had a chance 2 to one, analyze the report in more detail and two, to continue 3 and complete some of the analyses that we have underway. 4 I have to say quite frankly that in providing the 5 tow comments which I will ask Ken to do it may appear as 6 though we are being defensive or non-responsive and that is 7 not the case. I think the nature of the study that the IIT 8 conducted under the time limits that they had put them in a 9 certain light and there is a certain pressure to get the 10 report out. 11 I do share your dismay if you like in the fact that 12 the report does have a negative tone to it. However, 13 Mr. Martin in giving the oral report today did make some 14 positive statements. 15 I think in part some of the negativism of the report 16 may be due in fact to the fact that there is incomplete 17 information available at the time that the report was 18 completed either incomplete or based upon a subjective point 19 of view or perhaps some misleading information that may have 20 been available 21 I would like to ask Ken Baskin to comment on a few 22 of the findings and conclusions that are in the report and 23 that were on the screen so as to give you a flavor of what we 24 have been able to uncover and this is just a cursory review of 25 the findings that we have seen in the last few hours. i
92 1 Thank you. 2 CHAIMMAN PALLADINO: I should clear up. My dismay 3 was not with the negativism of the report but rather the facts 4 that the report uncovered and the fact that the facts were 5 there. 6 COMMISSIONEM ASSELSTINE: I agree. That is right.
? CHAIRMAN PALLADINO: Go ahead. Now this didn't mean 8 that I wasn't looking at the good points, also. But 9 nevertheless, even beside those good points we had a very 10 severe, very significant event take place that needs very 11 careful study and careful attention and appropriate corrective 12 action before the plant operates.
e 13 MR. PAPAY: I couldn't agree more. 14 MR. BASKIN: I wonder 11 it would be possible, 15 Mr. Chairman, 11 I could since we have had to put together 16 these remarks very rapidly and it is kind of disorganized if I 17 could borrow somebody's seat at the table here. 18 CHAIRMAN PALLADINO: Sure. 19 MR. STELLO: Please, take mine. l ( l 20 COMMISSIONER BERNTHAL: Most people wouldn't want a 21 seat at this tablel 22 (Laughter.) 23 MR. BASKIN: Under normal circumstances, I wouldn't i 24 ask for it but like I say, the organization nere is somewhat I 26 lacking in terms of our preparation this morning having had
93 1 just a very short time to review the report. ( 2 Let me emphasise what Larry Papay said. There is no 3 question in our mind that we consider what happened very, very 4 significant, that there are problems that have to be addressed 5 and will be addressed by the company and that changes in many 6 cases probably have to be made. The extent of those changes
? we certainly don *t know at this point in time but they have to 8 be made.
9 Again let me say I think it is ditticult in a 10 situation like this for us to comment without people interring 11 that we are being defensive or without people interring these 12 tellows don't understand, they don't have the message but that 13 is not our intent. 14 Our intent is to try and I think provide a little 15 balance here. With that, let me just make a few comments. 16 Let me say one other thing. The IIT process has a very 17 significant negative impact on the company being visited and 18 Vic mentioned that at some point in time they would like our 19 comments on how the process might be improved and we certainly 20 plan on doing that because we think it is important. 21 I would also like to comment that Tim Martin and the 22 tellows on the team, we feel did a fine job under the 23 constraints and ditticulties they had to work under so I am 24 not in any way here attempting to criticise the individual 25 members. We think they performed their job as well as was
94 1 possible. 2 Let me make a few specific comments on some at the 3 findings again just to put things in perspective and we will 4 certainly welcome and look forward to a time when we can come 5 back and discuss this in more detail 6 One of the findings, number 17, in essence says that 7 our process for evaluating events is not thorough and 8 systematic. That is the essence of the comment which you 9 gentlemen were discussing. 10 I guess we are chagrined to see that and we strongly 11 disagree with that. If you look at this particular event, for 12 example, one of the problems that a company faces is we have 13 the IIT team come down on site and make a very, very large 14 number of requests for information which is necessary for 15 their process. I am not knocking that. 16 In summary, we prepared roughly 25 comprehensive 17 white papers, 180 various document packages which responded to 18 over 150 specific questions in writing. This ended up as 19 thousands of pages of information that was given to the team. 20 Clearly the ottort that is involved in responding to 21 this interferes with our own investigation ettort as to the 22 root cause. There is no question about that. We nevertheless 23 had ettorts going on which were not in all cases completely J4 discussed with the IIT. 25 The point is the fact that they may have been ahead
1 l 95 1 1 of us and asked certain questions that we didn*t have answers 2 to is largely as a result of the process we were being 3 subjected to. 4 In addition, Tim and again rightly so would have 5 theories or questions and would ask various people in the 6 plant and that person may not have agreed or may not have 7 looked into it closely enough but that doesn't mean that was 8 the company's or the plant's position at that point in time. 9 That means it was one individual's position. Maybe the 10 process had not gone through the normal process before we had 11 been able to write it all That is a significant factor. 12 Let me mention the comment in here, number 14, that 13 indicates we delayed entry into the tech spec action 14 statements due to reluctance to shut the plant down, 15 Here I think we have a difference of opinion with 16 the team and it may be related to them not having all of the 17 tacts in this matter. The procedure for tracking down a fault 18 as existed is by intent quite general to leave the operator 19 some flexibility to do things. 20 The operator in his process went through and shut 21 ott all of the various pieces of equipment that did not 22 require a load reduction and did not discover the fault. 23 The next step would have been to shut att the 24 toedwater pump wnich did require a load rejection. While unit 25 load was botng reduced in order to do something, we then made
i 1 96
~
1 the transfer of that bus over to the other bus that was talked 2 about earlier. 3 In hindsight we think wnat we did was very proper 4 and very correct and in accordance with procedures and agajn I 5 don't want to get into technical details now but in looking at 6 this right now today they would do the same thing again 11
/ it happened tomorrow that we did at that time.
8 The third item I want to mention, a lot of 9 discussion on check valve testing and the tact that it is not 10 adequate to determine problems and valve tailures and that 11 type of thing. That is an absolutely true statement. There 12 is no question about it 13 The typical inservice testing does not detect the 14 sort of failures that would be detected. However, that 15 situation is certainly not unique to San Onotro. That 16 situation exists to the best of my knowledge in every nuclear . 17 plant in this country and in most of them overseas. 18 CHAIRMAN PALLADINO: That doesn't help the 19 situation. Excuse me, let me listen. 20 MR. BASKIN: My point is and you are right, it 21 doesn't help the situation and there is a problem there that 22 h .t s to be solved but I think it is important trem our 23 standpoint that you and us and the press understand that that l 24 is not some big deticiency at San Onotre. That is a problem l l 25 that exists in the industry that needs to be addresed. i l i
I 97 1 CHAIRMAN PALLADINO: Excuse me. There is a problem 2 at San Onofre with check valves. 3 COMMISSIONER ASSELSTINE: Not unique perhaps. 4 CHAIRMAN PALLADINO: I don't know whether it is
$ unique.
6 MR. BASKIN: There is a problem at San Onotre with
? check valves.
8 CHAIRMAN PALLADINO: But the tact that you think 9 that there may be problems elsewhere doesn't chango the 10 situation. 11 MR. BASKIN: I didn't mean to inter there may be 12 problems elsewhere. My point is the method of testing check 13 valves at San Oncire that did not detect these problems is the 14 same method that is used elsewhere. That is the point I was 1$ trying to make. 16 CHAIMMAN PALLADINO: We ought to discuss that some 17 more when we meet. 18 MR. BASKIN: Fine. The last comment I would make in 19 the interest of brevity here is that there were comments on 20 the maintenance records and the difficulty with the 21 maintenance records and certainly some of the records that the 22 team was looking for go back to the early and mid-1970's and 23 I would agree that the records that exist for that time trame 24 are certainly not of the standard we keep today and lack a lot 25 of detail and all of that.
1 1 The distinction I wanted to make is the recent 2 records we think are as good as one will find anywhere tram 3 the st%ndpoint of completeness and ability to understand what 4 took place. 5 One last comment -- 6 CHAIRMAN PALLADINO: Incidentally, that might be 7 worth looking at, more recent records. 8 MR. BASKIN: Somebody asked and I am sorry . torgot 9 who what the last SALP ratings were for this plant which are 10 as about nine months ago, I believe. Don't hold me to that. 11 In the maintenance area the SALP rating was one and in the 12 surveillance area, the SALP rating was two. 13 COMMISSIONER ASSELSTINE: Thank you. 14 COMMISSIONER BERNTHAL: Sounds like we screwed up. 15 (Laughter.) 16 MR. BASKIN: Unless there are questions that any of 17 you have, that would be all that we have at this time. 18 CHAIMMAN PALLADINO: Thank you very much. 19 COMMISSIONER BERNTHAL: I want to make a comment 20 because it is true that this had a rather negative tone to it 21 but you should understand that it is the job of the team to 22 find things that are wrong primarily and not to applaud the 23 things that may or may not be right 24 I will say that I think any resemblance between 25 Davis Bessie and Toledo Edison or the Toledo Edison situation
.m..._- _ _ - _ _ _ _ _ . _ _
l 1
. . I 99 i i
1 1 prior to Davis Bessie and the Southern California Edison 2 operation is coincidental. I think that the strength and 3 depth and capability of the programs of this utility are not 4 to be compared with what we have come to understand existed at 5 Toledo Edison. l 6 Toledo Edison, of course, has taken positive steps 7 since that time. I think here we have a plant with some 8 components and some problems. I think we do have a capable 9 utility here that I trust will take the necessary actions to 10 see to it that this p a r t i c*u l a r older plant gets a thorough 11 going over. i 12 I think you all will agree that it is clear that 13 that needs to be done. 14 MR. BASKIN: Yes. We don't dispute that. It I may 15 maybe help the statt save some work, you commented, 16 Commissioner Bernthal, about the discussion we had had on tech i ! 17 spec and tech spec changes and the large number of outstanding j 18 changes. 19 COMMISSIONER BERNTHAL: Right. 20 MR. BASKIN: While that situation has improved it l 21 still exists. However, that is essentially totally related to 22 San Onotre-2 and 'J and is not related to this issue. 23 COMMISSIONER BEHNTHAL: License amendments, I really , 24 misspoke wnen I said tech specs. It was license amendments i 25 per se that was the issue that was raised. I remember the 4
. . - . . . . _ - . _ . , . , - . , _ _ . . __ , . . , _ . _ . - , _ _ , , - - _ . _ _ _ , - - . - . ,,,-~__,___-.-,c._ - . _ . = _ . _ . , - - , _ _ _ _ _ _ ~ . . . . - _ . _ . _ _ , . _ -.y. .
1 number was 70 and I believe it was for San Onotre-1 or am I 2 wrong? 3 MR. BASKIN: There were a large number of license 4 amendments for unit-1 I trankly do not recall the number. 5 It was large. One of them was involved with the inservice 6 testing program that was discussed earlier that began -- 7 excuse me, discussions began in 1977 and it carried out. 8 My point is I don *t believe it is that large. I 9 think the larger number we looked at related to units-2 and 3 10 and the smaller number and what was on unit-1 was of less 11 significance than on units-2 and 3. 12 COMMISb10NEN BERNTHAL; All right. 13 COMMISSIONEM ASSELSTINE: I have just ons question, 14 Joe, and that would be Ken, do you have a rough time estimate 15 for when you think another meeting would be useful, give you a 16 chance to look at the IIT report, get a handle on wnst you all 17 want to do. 18 MR. BASKIN: My inclination would be to ask to come 19 back in like ten days to two weeks. 20 COMMISSIONER ASSELSTINE: That short? 21 MM. BASKIN: Yes, I think we can be ready that 22 soon. I think we need to do it quickly. Depending on your 23 schedule or whoever's schedule, that sort of time frame would 24 be satisfactory for us, 25 CHAIHMAN PALLADINO: I think agenda planning is
a . 101 1 tomorrow so we can take a look at it then. 2 COMMISSIOf4ER ASSELSTINE: Good. 3 CHAIMMAN PALLADINO: Thank you very much. We really 4 appreciate your statement. 5 MR, B A S Y.I N : Thank you, Mr. Chairman. 6 CHAIRMAN PALLADINO: Now I wonder it we could have 7 Just a briet status report on Mancho Seco. 8 COMMISSIONER ROBERTS: Very briet 9 MH. HELTMES: Yes, sir. I would be pleased to do 10 so. 11 CHAIRMAN PALLADINO: The last one I asked for to be 12 two minutes took 13. 13 MR. HELTMES: Yes, sir. I would' be pleased to give 14 you a status report of the Mancho Seco IIT. 15 As you know there was a severe over-cooling to transient at Mancho Seco the say after Christmas on the 26th 17 of December. At that time the event was discussed thoroughly 18 between headquarters statt and the region personnel and has 19 decided to respond by an AIT, by an Augmented Inspection Team, 20 using transortption of interviews and also using some 21 quarantine of equipment directly involved in the event. 22 After the AIT was on site for about three days we 23 learned more about the event and it was decided at that time 24 to upgrade the investigation to an IIT An IIT respcaded and 25 was on site for about ten days interviewing people, looking at
102 1 the troubleshooting procedures and constructing or developing 2 a sequence of events. 3 The team returned to headquarters. That is where 4 they are right now. 5 COMMISSIONER BERNTHAL: Is an augmented team not 6 integrated? 7 (Laughter.) 6 MM. HELTMES: The augmented team will be headed by 9 regional personnel primarily and it is a region directed 10 investigation. The team is currently in Bethesda. They are 11 analyzing the results and interviewing some stati They 12 intend to go back to the site next week to look at the trouble 13 shooting activities to get the r'e s u l t s of the latest 14 information available. They will come back ti. e n and complete 15 their report and the schedule currently calls for its issuance 16 on the 14th of February. 17 COMMISSIONEN ASSELSTINE: Do we have a meeting set 18 up on that one as well? 19 MR. HELTMES: I can't confirm that. I know I have 20 had discussions witn our people about setting up a meeting the 21 week after the 14th. 22 COMMISSIONER ASSELSTINE: All right. Good. I was 23 going to suggest that in the next instance it might be better 24 to have a few more days in bwtween when we get the report and 25 when we have the meeting.
103 1 CHAIMMAN PALLADINO: Yes, it would be very helpful 2 COMMISSIONEH ASSELSTINE: 1 think most of us skimmed 3 through pretty fast 4 MM. STELLO: We make the team work hard and make the 5 team work over Christmas and holidays and weekends. 6 CHAIRMAN PALLADINO: We appreciate that.
? MR. STELLO: We need to be willing to take that 8 extra--
9 COMMISSIONER ASSELSTINE: There is a certain 10 digestive factor though. 11 MM. STELLO: I appreciate that. 10 CHAIMMAN PALLADINO: We appreciate the hard work 13 that you put into this' matter. Nevertheless, I think it does 14 help it we have a little bit more time to absorb this 15 information. to MR. STELLO: We will do our best. Finally, let me 17 say with respect to where we are on Rancho Seco, we are 18 viewing this as an important issue with the B&W plants. We
.9 have been meeting with the owners groups and we are not going 20 to wait until the report comes out. We are going to continue 21 to open up our discussions with them and get things moving.
20 How far we need to go will depend on how events unfold in the 23 next month or so. 24 COMMISSIONER ASSELSTINE: One quick question, Joe, 25 11 I could on check valves, have you all sent out anything
104 1 yet? 2 MR. STELLO: Yes. 3 CCMMISSIONEM ASSELSTINE: Information notice or 4 something? You have them out already?
$ MR. STELLO: Yes. I think very shortly atter that 6 it was to get something out.
7 COMMISSIONEN ASSELSTINE: Great. 8 MM. STELLO: I think we want to probably take as 9 soon as this NUMEO comes out and send it around as well and to the team recognized that as the major issue and I certainly 11 can't disagree one bit and it is the one area that we want to 12 tocus attention on quickly and not that we want to dismiss the 13 other issues 'but that one I want to get out and get something 14 going very, very quickly, 15 COMMISSIONER ASSELSTINE: Good. 16 CHAIRMAN PALLADINO: I would like to thank the statt 17 tor a superb presentation. Tim, special commendation to your 18 presentation and the extent of knowledge that you displayed 19 about this is very impressive and I think it certainly gives 20 me a high degree of confidence that things are being looked 21 at. 22 MK. MARTIN: I had a good team. 23 COMMISSIONEH ASSELSTINE: Very nico job. 24 CHAlWMAN PALLAu!NO. Any other comments? 25 COMMISSIONER ZECH: Only that I would agree,
1 l .. . 1 105 1 CHAIRMAN PALLADINO. Thank you. We will stand 2 adjourned. 3 (Whereupon, the Ccmmisston meeting was ad3ourned at 4 12:20 o' clock p.m., to reconvene at the call of the Chair.) 5 0 7 8 9 10 11 12 13 14 15 10 17 18 19 20 21 22 23 24 25
1 CERTIFICATE OF OFFICipL PEPORTER 2 3 4 5 This is to certify that the attached proceedings 6 before the United States Nuclear Regulatory Commission in the 7 mattee o(. OC1HISSION MEETING G 9 Name of Proceeding: Briefing on San Onofre and Status of Rancho Seco (Public Meeting) 10 11 Docket No. 12 Place: Washincton, D. C. 13 Date: Wednesday. January 22, 1986 14 15 were held as herein appears and that this is the original 16 transcript thereof for the file of the United States Nuclear 17 Regulatory Commission. 13 l gg (Signature) , %g' y (Typed Name of Reporter) Lynn Nations j 20 21 22 23 Ann Rilesj & Associates, Ltd. 24 l 25 L
( . . COMMISSION BRIEFING LOSS OF POWEP AND WATER HAMMER EVENT SONGS ] NOVEMBER 21, 1985 T. T. MARTIN 488 1280 JANUARY 22, 1986
SAN ONOFRE,ilNIT 1 (SONGS-1) OPERATED BY SOUTHERN CALIFORNIA EDIS0N LOCATED SOUTH OF LOS ANGELES NEAR SAN CLEMENTE, CALIFORNIA WESTINGHOUSE 3-LOOP PWR LICENSED 1967 1337 MWT / 450 MWE SPHERICAL STEEL CONTAINMENT WITH CONCRETE . ENCI.0SilRE BilllDING ELECTRIC MAIN FEEDWATER PUMPS ALSO FilNCTION AS SAFETY INJECTION PUMPS
- q kNUkR 22 1986
.__________7
. ~.
SAN ON0FRE, UNIT 1 CONT'D 1 TURBINE AND 1 ELECTRIC AUXILIARY FEEDWATER PUMP 1 IMMEDIATE AND 1 DELAYED ACCESS OFFSITE POWER SOURCE . DIESELS START, BUT DO NOT AllT0MATICALLY LOAD ON LOSS OF POWER 4 l
- g j, ' kNUkR 77 19
EVENT OVERVIEW LOSS OF IN-PLANT AC POWER IN0PERABLE FEED PUMP CHECK VALVE LEADS TO RUPTilRE OF CONDENSATE SYSTEM COMP 0NENT LOSS OF FEEDWATER MillTIPLE INOPERABLE FEEDWATER CHECK VALVES All0W BACI' FLOW FROM All STEAM GENERATORS WATER liAMMER IN B FEED LINE CAllSES PIPING, Pl. PING SilPPORTS AND COMPONENT DAMAGE DAMAGED FEEDWATER CHECK VALVE DEVELOPS SIGNIFICANT STEAM-WATER LEAK - PLANT SHUTDOWN AND C00LDOWN COMPLETED SAFELY'(.
.? T. T. MARTIN, 488 1280 JANUARY 22, 1986
, ~,
TEAM ESTABLISHED BY ED0 ON NOVEMBER 22, 1985 IN CONFORMANCE WITH COMMISSION APPROVED INCIDENT INVESTIGATION PROGRAM CllARTER DETERMINE WHAT HAPPENED IDENTIFY PROBABLE CAUSES OF WHAT HAPPENED MAKE APPROPRIATE FINDINGS AND CONCLilSIONS.T0 FORM BASIS FOR POSSIBLE FOLLOW-0N ACTIONS
- (
T. T. MARTIN, 488 1280 JAN[lARY 22, 1986
t a TEAM MEMBEPS 6 Til0 MAS T. MARTIN, TEAM LEADER MATTHEW CitiRAMAL WILLIAM G. KENNEDY WAYNE D. LANNING ALECK W. SERKIZ ! STEVEN K. SHOPE T. T. MARTIN, 488 1280
. JANUARY 22, 1986
FACT FINDING METH0D0L0GY INTERVIEWS AND MEETINGS PLANT DATA PERSONNEL LOGS QUARANTINED EQUIPMENT OBSERVATIONS , STATUS REPORTS ,
- (
T. T. MARTIN, 488 1280
,[ JANUARY 22, 1986
1 SE0llENCE OF EVENTS INITIAL CONDITIONS SMALL SALTWATER LEAK INTO MAIN CONDENSER UNIT OPERATING AT 60 PERCENT POWER STEAM GENERATOR BLOWDOWN ABOUT 100 GPM/ STEAM GENERATOR CRITICAL FUNCTION MONITOR SYSTEM DISABLED ELECTRICAL GROUND TR0llBLESH00 TING IN PROGRESS ELECTRIC PLANT IN UNilSUAL All.GNMENT T. T. MARTIN, 488 1280 JANUARY 22, 1986
SE0llENCE OF EVENTS T=0 (MIN) AUXILIARY TRANSFORMER C ISOLATES (PARTIAL LOSS OF AC POWER) T=0+ ENS PHONE RINGS FEEDWATER PUMP CllECK VALVE FAILS TO CLOSE FLASH EVAPORATOR TilBE RUPTilRES DIESEL GENERATOR 2 STARTS, BilT BY DESIGN DOES NOT LOAD T=1/3 OPERATORS TRIP REACTOR AND llNIT GENERATOR (TOTAL LOSS OF INPLANT A/C POWER) CONTAINMENT ISOLATES DIESEL CENERATOR 1 STARTS, BUT BY DESIGN DOES NOT LOAD TilRBINE-DRIVEN AUXILIARY FFEDWATER PUMP RECEIVES START SIGNAL F0llR ADDITIONAL FEEDWATER CHECK VALVES IN0PERABLE
. ,[ T. T. MARTIN, 488 1280 JANIIARY 22, 1986
SEQUENCE OF EVENTS T=1/3+ FEEDWATER LINES BEGIN TO EMPTY STEAM GENERATORS BEGIN TO LOSE INVENTORY VIA FAILED CHECK VALVES AND RUPTilRED TilBE IN FLASH EVAPORATOR LOSS OF VOLTAGE AUTOMATIC TRANSFER SCHEME INITIATED ALARMS INDICATE SAFETY INJECTION ACTilATION T=3 FIRE TRUCK ARRIVES ON SITE h T. T. MARTIN, 488 1280 JANUARY 22, 1986
SEQUENCE OF EVENTS T=fi OPERATORS ATTEMPT TO RESTORE POWER TIIRBINE-DRIVEN AFW FLOW AB0llT 130 GPM/ STEAM GENERATOR T=4+ OPERATORS RESTORE POWER ON FIFTH ATTEMPT MOTOR-DRIVEN AFW PllMP STARTS INCREASING FLOW TO 155 GPM/SG ATMOSPHERIC STEAM DlJMPS OPEN OPERATORS CLOSE FEEDWATER ISOLATION VALVES RADIATION MONITOR ALARMS RESET (S/G BLOWDOWN RE-ESTABLISHED)
' T. T. MARTIN, 488 1280
. JANllARY 22, 1986
i l SEQUENCE OF EVENTS T=6 RCS PRESSURE, TEMPERATURE AND PPESSilRIZER LEVEL INDICATE RAPID C00LDOWN T=7 OPERATORS START CHARGING PLIMP SECOND PUMP AtlTO STARTS LATER T=10 REACTOR COOLANT PUMP B STARTFD T=11 OPERATORS TERMINATE (10 SECONDS) AND RE-ESTABLISH AUXILIARY FEEDWATER FLOW AT 40 GPM/SG T=]5 UNUSUALEVENTDECLAREDONSITElNOTTONRC 8 T. T. MARTIN, 488 1280 JANUARY 22, 1986
SE0llENCE OF EVENTS T=16 WATER HAMMER OCCURRED T=18 RCP B THRilST BEARING HIGH TEMPERATURE ALARM T=19 PLANT EQUIPMENT OPERATOR INFORMS CONTROL ROOM 0F "STEAMLINE BPEAK" T=25 CONTAINMENT COOLING PE-ESTABLISHED T=29 OPERATORS SECURE DIESEL GENEPATORS OPERATORS SECilRE LilBE OIL RESFRVOIR F0AM SYSTEM 4
.g
,[ kNkP 22 19 6
SE0llENCE OF EVENTS T=33 PCP A STARTED T=36 RCP C STARTED STEAM GENERATOR LEVELS OFF-SCALE LOW T=37 RCP B STOPPED AFW FLOW TO A AND C INCREASED T0 70 GPM/SG TO ESTABLISl! RAPID Bi1T ACCEPTABLE C00LD0WN T=39 STEAM GF.NERATOR BLOWDOWN DISCOVERED AND TERMINATED T=40 STEAM GENERATOR A AND C LEVEL.S ONSCALE ATTEMPTS BEGIN TO IDENTIFY STEAM LEAK T. T. MAPTIN, 488 1280
.' JANilARY 22, 1986
SE0llENCE OF EVENTS T=90 (MIN) OPERATORS UNABLE TO START 1 CIRCllLATING WATER PilMP T=3 (HR) ENTERED MODE 4 T=4 OPERATOR OVERRODE PHP INTERLOCK T=5 llNUSUAL EVENT TERMINATED T=6 FEEDWATER LEAKAGE ISOLATED T=]0 ENTERED MODE 5 , a T. T. MARTIN, 1i88 1280
,- JANllARY 22, 1986 0
SE0llENCE OF EVENTS NOVEMilER 22 CONTAINMENT ENTRY FOUND EVIDENCE OF WATER HAMMER DAMAGE t 8 1 ' T. T. MARTIN, 488 1280
,- JANilARY 22, 1986
PERSONNEL PERFORMANCE EVAL.UATIONS OPERATOP ERRORS FAILED TO F01.1.0W APPROPRI ATE PROCEDl!RES WHEN TP0llBLESH00 TING ELECTRICAL GPOUND DIFFICULTY IN RE-ESTABLISHING INPLANT POWER INADVERTENTLY RE-ESTABLISHED STEAM GENERATOR BLOWDOWN DID NOT RESET FOX III COMPUTER STA PERFORMANCE GOOD EMERGENCY COORDINATOR PERFORMANCE ERRORS WERE NOT EFFECTIVE NPC-LICENSEE COMMilNICATIONS.1, T. T. MARTIN, 488 1280
', JANilARY 22, 1986
~
E0llIPMENT PROBLEMS PROBABLE ! ITEM NATURE OF FAllllRE ROOT CAUSE COMMENTS
- 1. POWER SUPPLY CABLE GROUND FAULT WATER INLEAKAGE ANALYSES ONG0lFG
- 2. FLASH EVAPORATOR tlNIT TUBE /SHELL RUPTURED DVERPRESSURIZED DIJE N0 1.0NGER USED TO FAILED CHECK VALVE
- 3. SAFETY INJECTION SPURIOUS ALARM LOSS OF POWER DESIGN INADE011ACY ANNilNCIATOR '
4, SAFEGUARD LOAD INDICATED SAFETY NOT IDENTIFIED ANALYSES ONG0ING SEQUENCING SYSTEM INJECTION ACTUATION
- 5. I.0SS OF VOLTAGE AUTO FAILED TO REAl.IGN NOT IDENTIFIED ANALYSES ONG0ING TRANSFER SCilEME CIRCUIT BREAKERS TO RESTORE POWER 1,
,[ T. T. MARTIN, 488 1280 JANUARY 22, 1986
. ~.
E0lllPMENT PROBLEMS CONT'D PROBABLE ITEM NATllRE OF FAILURE P00T CAllSE COMMENTS
- 6. FOX III COMPUTEP NO RECORDED DATA POWER INTERPllPTION RESET RE0llIRED BEFORE/AFTEP TRIP
- 7. TilRBINE RUPTURE RUPTIIRED OVERPRESSURIZED DUE EXPECTED DISKS (4 0F 8) TO LOSS OF POWER
- 8. EMERGENCY NOTIFICA- SPURIOUS RINGS NOT IDENTIFIED CANNOT REPRODllCE TION SYSTEM
- 9. RCP THRUST BEARING llIGH TEMPERATURE FAILED DETECT 0P ALARM 10 CilECK VALVE STUDS STRETCHED WATER HAMMER FWS-378 BODY TO BONNET l.EAK .
- T. T. MARTIN, 488 1280
', JANUARY 22, 1986 l
EQUIPMENT PROBLEMS CONT'D PROBABLE ITEM NATURE OF FAILURE ROOT CAllSE COMMENTS
- 11. CllECK VALVE NUT MISSING NOT IDENTIFIED EVALUATION ONG0ING FWS-345 DISC SEPARATED FROM llINGE ARM
- 12. CilECK VALVE NUT MISSING NOT IDENTIFIED EVAltlATION ONG0ING FWS-346 DISC SEPARATED FROM HINGE ARM
- 13. CHECK VALVE NUT LOOSE NOT IDENTIFIED EVALUATION ONG0ING FWS-398 STUCK-0 PEN ,
- 14. CHECK VALVE NUT LOOSE NOT IDENTIFIED' NOT PINNED FWS-438 STUCK-0 PEN EVAltlATION ONG0ING
'1,
.' T. T. MARTIN, 488 1280 JANUARY 22, 1986
E0lllPMENT PROBLEMS CONT'D PROBABLE ITEM NATURE OF FAllllRE P00T CAllSE COMMENTS
- 15. CHECK VALVE NUT LOOSE NOT IDENTIFIED NOT PINNED FWS-439 STilCK-0 PEN EVALUATION ONG0ING
- 16. FLOW CONTROL VALVE BROKEN YOKE WATER HAMMER INERTIA FCV-457 BENT STEM
- 17. B STEAM GEPERATOR CRACKED WATER HAMMER EVAll!ATION FEEDWATER LINE BENT ONG0ING DENTED BEING PEMOVED
, MOST SilSCEPTIBLE PIPING
- 18. FEEDWATER LINE DAMAGED WATER llAMMEP EVALUATION SllPPORTS 8 SNUBBERS '
ONG0ING
'1,
.- T. T. MAPTIN, 488 1?80
, JANilARY 22, 1986
! EQUIPMENT PP0BLEMS CONT'D i l i PROBAPLE ITEM NATURE OF FAllllRE P00T CAllSE COMMENTS
- 19. AUXILIARY FEEDWATER DISPLACEMENT WATER HAMMER ,
LINE SUPPORTS t i
- 20. CONTAINMENT SPHERE SMAll CPACK-LIKE NOT IDENTIFIED EVALUATION INDICATIONS ONG0ING
- 21. SECURITY ACCESS SECURE SAFEGilARDS N0 SAFETY / SAFE-CONTROLS INFORMATION GUARDS INTERFACE PROBLEM
~
T. T. MARTIN, 488 1280
,- JANllARY 22, 1986
PRINCIPAL FINDINGS AND CONCLilSIONS Tile EVENT WAS SIGNIFICANT ALL INPLANT AC POWER WAS 1.0ST FOR FOUR MINUTES ALL STEAM GENERATOR FEEDWATER WAS LOST FOR TilREE MINilTES A SEVERE WATER HAMMER WAS EXPERIENCED IN THE FEEDWATER SYSTEM CAllSED A LEAK DAMAGED PLANT E0!!IPMENT CHALLENGED THE INTEGRITY OF Tile IILTIMATE IIEAT SINK , ALL INDICATED STEAM GENERATOR WATER I.EVELS DROPPED BELOW SCALE THE REACTOR COOLANT SYSTEM EXPERIENCED AN ACCEPTABLE BtlT 11NNECESSARY C00LDOWN TRANSIENT T. T. MARTIN, 488 1280 JANtlARY 22, 1986
SIGNIFICANT FINDINGS (NOT NECESSAPILY IN ORDER OF TilEIR SIGNIFICANCE)
- 1. Tile PRIMAPY CAllSE FOR THE WATER liAMMEP. IN THE FEEDWATER PIPING WAS THE FAllllPE OF MilLTIPl.E CllECK VALVES IN Tile FEEDWATER SYSTEM. THESE FAILIIRES PERMITTED THE PIPING TO EMPTY AND Fill. WITH STEAM BEFORE Tile MOTOP-OPERATED FEEDWATER IS01.ATION VALVES WEPE CLOSED, Al.Til0 UGH THE STEAM CONDENSATION-INDUCED WATER llAMMER OCCURRED IN ONLY ONE FEEDWATER LINE, Tile P0TENTI AL EXISTED FOR WATER HAMMEP TO OCCUR TilROUGH0llT THE SAFETY-RELATED PORTIONS OF THE FEEDWATER SYSTEM.
- 2. Tile FAILilRES OF THE FIVE CHECK VALVES IN THE FEEDWATER SYSTEM PROVIDED A MECHANISM FOR P0TENTIAL COMMON MODE FAILURE OF THE HEAT SINK PROVIDED BY THE TilREE STEAM GENERATORS. THE FAILED CHECK VALVES PERMITTED llIGli PRESSURE STEAM AND WATER FROM Tile STEAM GENERATORS TO Fl.0W BACK TO Tile LOW PRESSlJRE CONDENSATE SYSTEM; THE BACKFLOW CARRIED WITH IT Tile AUXILLARY FEEDWATER FLOW NECESSARY TO MAINTAIN THE IlEAT SINK PROVIDED BY THE STEAM GENERATORS. OPERATOR ACTIONS WERE NECESSARY TO STOP Tile BACKLEAKAGE AND PREVEflT A MORE SEPI0llS SEQUENCE OF EVENTS.
- 3. LONG HORI7ONTAL RUNS OF FEEDWATER PIPING WITil Tile POTENTI AL FOR V0IDING ARE PARTICULARLY SUSCEPTIBLE TO DESTRUCTIVE STEAM CONDENSATION.-INDilCED WATER HAMMERS.
FilRTilER, OPERATORS ARE NOT PROVIDED THE MEANS F0P DETECTING Tile V0IDING 0F TilESE LINES OR GIVEN GillDANCE ON APPROPRI ATE WAYS TO DEAL WITil THE SIYNATION. DESIGN OR PROCEPURAL CliANGES MAY BE WAPRANTED. , T. T. MARTIN, 1188 1280 JANUARY 22, 1986
SIGNIFICANT FINDINGS CONT'D
- 4. THE FLASH EVAPORATOR FAILED WilEN OVERPRESSURIZED BY Tile DISCllARGE FLOW 0F AN OPERATING FEEDWATER Pl!MP DllE TO THE PARTIAL LOSS OF POWER AND A STilCK OPEN FEEDWATER PUMP DISCilAPGE CHECK VALVE TilAT Sil00LD llAVE PREVENTF_D Tile BACKFLOW.
- 5. Tile TIMING 0F THE FIVE CliECK VALVE FAILURES C0lJLD NOT BE ASCERTAINED WITil CERTAINTY, Tile TEAM CONCLUDED THAT ALL CilECK VALVES IIAD FAILED PRIOR TO THE EVENT BECAllSE Tile MISSING PAPTS TO THE val.VES WERE NOT F0llND IN THE INSPECTED FEEDWATEP PIPING AFTER Tile EVENT, NOISE FROM Tile B STEAM GENERATOR FEEDWATEP PIPING, EVIDENT TO PLANT PERSONNEL SINCE JUNE 24, 1985, SilPPORTS THE CONCLilSION TilAT THE FEEDWATER CONTPOL STATION CHECK val.VE IN THE B FEEDWATER LINF llAD FAILED EARLIER. THE INSPECTION OF THE STEAM GENER-ATOPS 11AS NOT YET BEEN COMPLETED BY SCE.
- 6. Tile SilRVElli.ANCE PROCEDllRE FOR TESTING THE CHECK val.VES IN THE INSERVICE TESTING (IST) PP0 GRAM LACKED ADE0llATE METHODS AND OBJECTIVE ACCEPTANCE CRITERIA FOR DETER-MINING WilETilER CHECK VALVES ARE Cl.0 SED. TlillS, ALTHOUGli Tile CHECK VAI.VES HAD BEEN TESTED WITilIN Tile PAST YEAR, OPERATORS MAY llAVE MISINTERPRETED THE TEST RESilLTS.
FURTilERMORE, THE IST IS NOT DESIGNED TO DETECT DEVELOPING ' CONDITIONS THAT MAY LEAD STUD N!!TS. TO THE FAILURE OF THE CllECK VALVES; E.G., LOOSE DISKS AND.t
.' T. T. MARTIN, 488 1280
', JANUARY 22, 1986
SIGNIFICANT FINDINGS CONT'D
- 7. THE NRC HAD NOT COMPLETED ITS REVIEW 0F SCE'S INSEP.VICE TESTING PROGRAM. THE INITIAL PROGRAM WAS SUBMITTED IN SEPTEMBER 1977 AND REVISED IN ITS ENTIRETY ON JANUARY 24, 1984. DISAGREEMENT BETWEEN SCE AND NRC ON RESOLUTION OF CERTAIN OPEN ISSUES AND SCHEDULING PROBLEMS WITil NRC'S REVIEW HAVE SilBSTANTIVELY CONTRIBllTED TO THIS DELAY.
- 8. THE RES0LilTION OF THE UNRESOLVED SAFETY ISSUE, USI A-1, " WATER HAMMER," DID NOT SPECIFICALLY ADDRESS THE PREVENTION AND MITIGATION OF THE CONSEQUENCES OF CONDENSATION- INDUCED WATER llAMMERS IN FEEDWATER PIPING llPSTREAM 0F THE FEEDRING.
INTERVIEWS OF NRC STAFF INVOLVED IN RESOLUTION OF WATEP iAMMER ISSUES FAILED TO DEVELOP CITABLE REFERENCES, DECISIONS, OF DISCUSSIONS TilAT PROVIDED A BASIS FOR EXCLUDING FilRTilER CONSIDERATION OF FEEDWATER PIPING WATER llAMMER. HOWEVER, IN THE REGULATORY ANALYSIS OF Tile RESOLUTION OF USI A-1, THE STAFF ACKNOWLEDGED THAT ELIMINATION OF WATER HAMMERS IS NOT FEASIBLE, TilAT THE FREQUENCY OF WATER HAMMERS HAD BEEN SilBSTAtlTI Al.LY PEDUCED BY CHANGES IN DESIGN AND OPERATIONS, AND TilAT STilDIES OF WATER llAMMER llAD REVEALED A SIGNIFICANTLY LESSEP SAFFTY CONCERN THAN PREVI0llSLY llYPOTilESIZED . IT APPEARS TilAT FllRTilER CONSIDERATION OF WATER liAMMERS DilE TO MAIN FEEDWATER LINE VOIDING WAS NOT PilRSilED DilE TO A l.ACK 0F RE. PORTED OCCl]RRENCES IN ll.S. PLANTS.
.A
- T. T. MARTIN, 488 1280 JANilARY 22, 1986
SIGNIFICANT FINDINGS CONT'D
- 9. FRC'S RELIANCE ON "J" TilBES TO DELAY THE DEVELOPMENT OF CONDITIONS NECESSARY TO SIJPPORT STEAM GENERATOR WATER HAMMER IMPLICITLY ASSUMES THAT FEEDWATER CHECK VALVE INTEGRITY WOULD BE MAINTAINED TO PREVENT STEAM GENERATOP FEEDRING VOIDING, HOWEVER, CORRESPONDING REGULATORY RE0lilREMENTS TO ENSilRE THAT THESE CHECK VALVES PERFORMED THIS SAFETY FUNCTION WERE NOT PART OF THE RES0LilTION OF THE WATER llAMMER ISSilE.
- 10. THE ROOT CAUSE FOR THE LOSS OF POWER WAS A PHASE-TO-PilASE FAULT OF AN ELECTRICAL CABLE FROM AtlXILIARY TRANSFORMER C TO BilS 1C. THE l'NDERLYING REASON FOR Tile CABLE FAILURE HAS NOT YET BEEN DETERMINED; HOWEVER, IT APPEARS TilAT Tile CABLE MAY llAVE BECOME WETTED BY A LONG-TERM FLANGE LEAK FROM THE FEEDWATER SYSTEM, RUNNING AB0VE Tile CABLE TRAY.
- 11. Tile PLANT IS DESIGFED TO EXPERIENCE AN EXTENDED LOSS OF INPLANT AC POWER ON 1.0SS OF 0FFSITE POWER WITil00T SAFETY INJECTION, OPERATORS ARE REQUIRED TO RESTORE POWER FROM THE SWITCllYARD OR TO LOAD THE DIESEL GENERATORS TO RESTOR $ INPLANT POWER, SCE'S EMERGENCY OPERATING INSTRUCTIONS ON LOSS OF AC POWER LACK GUIDANCE ON HOW LONG OPERATORS CAN ATTEMPT TO RESIORE POWER FROM 0FFSITE SOURCES BEFORE THE DIESEL GENERATORS SHOULD BE LOADED FOLLOWING A LOSS OF INPLANT AC. POWER, OR 110W LONG Tile DIESEL GENERATORS CAN RilN UNLOADED WITil0llT OVERHEATING, IF THEIR AC-POWERED RADI ATOR FANS REMAIN DE-ENERG12ED. '
- T. T, MARTIN, 488 1280
', JANllARY 22, 1986
SIGNIFICANT FINDINGS CONT'D
- 12. THE STATION LOSS OF VOLTAGE AUTO TRANSFER SCHEME FOR ESTABLISHING THE DELAYED ACCESS TO 0FFSITE POWER MAY NOT HAVE FUNCTIONED AS DESIGNED, SCE EVAlllATIONS ARE CONTINllING.
- 13. Tile Mill.TIPLE SPURI0llS INDICATIONS EARLY IN Tile EVENT TilAT A SAFETY INJECTION ACTUATION llAD OCCURRED, ADDED TO THE CONFUSION OF Tile SITUATION AND UNNECESSARILY INCREASED THE BilRDEN ON THE OPERATORS. OPERATORS DIAGNOSED PLANT CONDITIONS AND APPROPRIATELY DIS-REGARDED THESE INDICATIONS. THE SAFETY INJECTION ANNUNCIATOR WILL ALWAYS INCORRECTLY ALARM OM A LOSS OF AC POWER, THIS IS A DESIGN DEFICIENCY. THE CAllSE OF THE SPUR 10llS INDICATION ON BOTH SAFEGUARD LOAD SEQUENCEP SYSTEM PANELS IS STILL UNKNOWN.
- 14. Tile OPERATING STAFF, WITH THE. CONCURRENCE OF MANAGEMENT, DID NOT F01.1.0W APPROPRIATE PROCEDilRES WHEN TROUBLESHOOTING THE ELECTRICAL GR0llND. THEIP ACTIONS UNNECESSARILY DELAYED ENTRY INTO TECHNICAL SPECIFICATION ACTION STATEMENT REQUIREMENTS THAT C0llLD RE0llIRE Pl. ANT SHUTDOWN,
- 15. ONCE Tile ELECTRICAL GR0llND WAS LOCATED ON Tile FEEDER FROM AllXILIARY TRANSFORMER C TO BilS 1C, THE OPERATORS DID NOT AGGRESSIVELY PURSUE ISOLATING THE AllXILI ARY TRANSFORMER. INSTEAD, TilEY OPTED TO LEAVE THE TRANSFORMEg ENERGIZED WilILE TECHNICI ANS PERFORMED INSPECTIONS THAT DID NOT REQllIRE Tile TRANSFORMER.'T0 BE ENERGIZED,
. T T. MARTIN, 488 1280 JANUARY 22, 1986
i SIGNIFICANT FINDINGS CONT'D 16. THE OPEPATOPS' ACTIONS AFTER THE TRANSFORMER TRIP PERE CONSISTENT WITH THEIR TPAINING. HOWEVER, IN THE TEAM'S JllDGMENT, SOME OPERATOPS, LACKED DETAILED PLANT KNOWLEDGE IN THE F0LLOWING AREAS: , CAUTIONS ASSOCIATED WITH PARALLELING TRANSFDPMERS PE0llIPEMENTS FOR RESETTING UNIT GENERATOR TRIPS THE PROCESS FOR OPERATING 220KV CIRClllT BREAKEPS EXPECTED INDICATIONS AND TIMING 0F.THE LOSS OF V0LTAGE AUTOMATIC TRANSFER SCHEME SETP0INTS FOR RESID11Al. HEAT REMOVAL SYSTEM PRESSilRE INTERLOCK l EXPECTED INDICATION AND MEANING OF LIGHTS ON SLSS SE0llENCEP PANEL.S OPERAPILITY OF DIESEL GENERATORS WITil AUXILIARY TRANSFORMER C' REACTOR C0ll j BYPASS BREAKERS REMOVED. THESE DEFICIENCIES MAY BE DUE TO INADEQUATE OPERATOR TRAINING AND/0P PROCEDllRES. A l . T. T. MARTIN, 488 1280 f ' JANUARY 22, 1986 9
SIGNIFICANT FINDINGS CONT'D
- 17. ON OCCASION, SOME SITE PERSONNEL Wil0 GENERALLY EVAlllATE PLANT DATA LACKED A SUFFICIENTLY IN0UIRING ATTITUDE. AS A RESULT, CERTAIN SIGNIFICANT INDICATIONS OF UNDERLYING REASONS FOR SYSTEM RESPONSE OR COMPONENT PEPFORMANCE WERE NOT DETECTED UNTIL BROUGHT TO THE ATTENTION OF SCE BY THE TEAM. IT APPEAPS TilAT SCE'S PP0 CESS FOR EVAltlATING AND FOLLOWING UP EVENTS MAY NOT BE SUFFICIENTLY TH0P0l!GH AND SYSTEMATIC TO ASSUPE THAT FAILED COMPONENTS ARE DETECTED AND ADE0llATELY EXPLAINED.
- 18. Tile STATUS OF THE STEAM GENERATOR BLOWDOWN SYSTEM IS NOT INDICATED IN THE CONTROL POOM. Tile REESTABLISHMENT OF BLOWDOWN WHEN Tile PADIATION MONITORS WERE RESET WAS NOT RECOGNIZED AND ADVERSELY CONTRIBilTED TO THE C00LDOWN OF Tile PEACTOR COOLANT SYSTEM AND TO THE DEI.AY IN REC 0VERING THE STEAM GENERATOR LEVELS.
1
- 19. DilRING THE LOSS OF ALL INPLANT AC POWER, SilFFICIENT INFORMATION WAS AVAILABLE IN THE CONTROL ROOM TO ENABLE TliE OPERATOPS TO FOLLOW THEIR PROCEDllPES AND ENSUPE PLANT SAFETY, HOWEVER, CONTROL ROOM GPEPATORS IIAD FAILED TO IIAVE THE TECHNICAL SUPPORT CENTEP COMPilTER RESET F0LLOWING ELECTRICAL GR0llND TR0llBLESH00 TING ACTIVITIES. THIS
! FAILURE DISABLED Tile COMPUTER'S ABILITY TO RECORD NEW PLANT DATA'AND THEREBY DENIED ! Tile OPERATORS ACCESS TO PRE-TRIP AND POST-TRIP TRENDS THAT W0llLD PAVE ASSISTED PEAL TIME AND POST-EVENT ANALYSIS AND EVALUATION. IIAD THE STATION BLACK 0UT BEEN OF LONGER i DitPATION, OR INVOLVED ADDITIONAL COMPLICATIONS, OPERATOR RESPONSES AND THE FUNCTIONS PROVIDEDBYTilETECHNICALSitPPORTCENTERCOULDHAVEBEENIfAMPEREDBYTHELACK0FTREND DATA. '
', T. T. MAPTIN,.1280 JANilARY 22, 1986
l SIGNIFICANT FINDINGS CONT'D
?0. STATION MAINTENANCE REC 0PDS APE INCOMPl.ETE, DIFFICULT TO LOCATE AND, WHEN AVAILABLE, LACK SUFFICIENT DETAIL TO DETERMINE WHAT WAS DONE.
- 21. THE SPURIOUS RINGING 0F THE NRC RED PHONE AT THE BEGINNING 0F THE EVENT HAS NOT BEEN EXPl.AINED, BUT IT DISTRACTED CONTROL ROOM PEPSONNEl. AND CONTRIBUTED TO THE CONFilSION IN THE COMMUNICATIONS BETWEEN SCE AND NRC.
- 22. ENS COMMllNICATIONS BETWEEN NRC AND SCE WERE NOT EFFECTIVE BECAUSE: (1) THE NRC DllTY OFFICER WAS NOT KNOWLEDGEABLE ABOUT THE UNIQUE DESIGN OF THE PLANT AND, THEREFORE, MISINTERPRETED OPERATOR RESPONSES TO QUESTIONS; (2) COMMUNICATIONS WITH THE PLANT WEPE INITI ALLY LIMITED BECAllSE STATEMENTS BY PLANT OPERATORS INCORRECTI.Y IMPLIED THAT SUFFICIENT PERSONNEL WERE NOT AVAILABLE TO SllPPORT THE ESTABl_ISHMENT OF AN OPEN LINE; (3) NRC ASKED LEADING OVESTIONS AND OPERATORS SOMETIMES DID NOT CORRECT, AND IN SOME CASES APPEARED TO CONFIRM, INACCURATE INFORMATION; (II) NRC QUESTIONS CHARACTER-ISTICALLY FOCllSED ON DETAILS RATHER THAN ON THE " BIG PICTURE"; (5) NPC CLUTTERED THE COMMllNICATIONS CHANNEL WITH REPETITIVE DISCUSSIONS AB0llT THE SE0tlENCE OF EVENTS AS ADDITIONAL NRC PERSONNEL CAME ON THE LINE TO Tile EXCLilSION OF OBTAINING NEW PLANT INFORMATION: (6) NRC RESIDENT INSPECTORS RELIEVED MORE KNOWI.EDGEABLE PLANT OPERATORS AS ENS COMMUNICATOPS AND REESTABLISHED THE COMMlINICATIONS'.AT A LOCATION REMOTE FROM REAL TIME PLANT INFORMATION:
AND,(7)PLANTOPERATORSFAIL['DTOINFORMTHENRC0FTHE DECLARATION OF AN UNllSilAL EVENT.
; T. T. MARTIN, 488 1280
', JANUARY 22, 1986
SIGNIFICANT FINDINGS CONT'D
- 23. TilERE WERE TWO MALFUNCTIONS OF Tile AllT0 MATED SECURITY ACCESS CONTROL EQUIPMENT; 110 WEVER, SITE PERSONNEL IMPLEMENTED APPPOPRI ATE PLANNED COMPEFSATORY MEASURES, TilERERY PRECI.UDING A SAFETY-SAFEGilARDS INTERFACE PROBl.EM, 2 11 . Ti1ERF WAS NO SIGNIFICANT RELFASE OF RADI0 ACTIVITY.
T T. MARTIN, 'I88 1280
. JANIJARY 22, 1986
e . CONCLilSION THE MOST SIGNIFICANT ASPECT OF Tile EVENT WAS THAT FIVE SAFETY-REl.ATED FEEDWATER SYSTEM CHECK VALVES DEGPADED TO THE P0 INT OF IN0PERABILITY DUPING A PEPIOD OF LESS TilAN A YEAR, WIT 40llT DETECTION, AND TilAT THEIR FAILURE JEOPARDIZED THE INTEGRITY OF SAFETY-RELATED FEEDWATER PIPING. THE ROOT CAllSES OF Tile CllECK VALVE FAILURES IIAVE NOT BEEN DETERMINED AND ARE STILL UNDER REVIEW BY SCE AND ITS CONTRACT 0PS POTENTI AL CONTRIBilTOPS TO Tills PROBLEM INCLl!DE INADEQUATE MAINTENANCE, INADEQUATE INSERVICE TESTING, INADE0llATE DESIGN, AND INADE0llATE CONSIDERATION OF THE EFFECTS OF REDUCED POWER OPERATIONS. 1
- T. T. MARTIN, f188 J280
,. JANilARY 22, 1986
'~ .
P0TENTIAL CONTRIBilTORS TO FAltllRES MAINTENANCE RECORDS FOR THESE VALVES WERE EITHER MISSING OR LACKED SPECIFICITY ON WHAT WAS DONE l INSERVICE TESTING RECORDS FOR THESE VALVES WERE INCONSISTENT; THE TESTING PROCEDURE WAS NOT RIGOROUS; THE TEST ACCEPTANCE CRITERIA WERE SUBJECTIVE; THE TESTING FREQUENCY WAS OPEN-ENDED: AND, THE TEST DID NOT ASSURE DETECTION OF THE FAILURES F0llND. TilESE CHECK VALVES AND VALVES OF SIMILAR DESIGN HAVE A HISTORY OF LIKE FAILURES REDUCED POWER OPERATIONS AT UNIT 1 ARE N0W ROUTINE BECAllSE OF STEAM GENERATOR TilBE PLUGGING AND SLEEVING, AND THE REDilCED FEEDWATER FLOW MAY HAVE INCPEASED THE SUSCEPTIBILITY OF CHECK VALVE COMPONENTS TO HYDRAULIC-INDUCED VIBRATION. T. T. MARTIN, 1188 1280
,- JAMllARY 22, 1986
NUREG-1190 Loss of Power and Water Hammer Event at San Onofre, Unit 1 on November 21,1985 U.S. Nuclear Regulatory Commission pem s, ! -...-,.r.____. , , _ _ _ _ . _ . - _,,,,-,,.,,,,,______,.___.r_ _
l 1 Z w w g E E C z O >- C
<we EO
<z wws am-w wo zzo m Oaz v 'n *
$zd O C, c wo
-.4...
i . . ..- -
'h -
1 . s j .
- g. *' y
_, \- <-
\- = ., * , ~ I. 3
( gw -
- h. * '
[z ,. az :< 1; - f< % a C$. s 3 : , . C .. 4
- 2 -. .
- g 7
- , %g' . .
-- *- y =
< . p. .w. . -. -
-2 C l ,
'F 6
.'[
- h.
- g
'A i
,,l.-
^
',A y
(
,.- .x -
s - l 1 1 5 y
' ,n
. m .
5 i - C
,, :s =
< '- E 1 4 %.% >
} c
-a O e" %% ~ g4 ,. - k :
'.'9.
", [ C w .- +
4( s: - . h. y '). 5 '. y ; ., ' - - , [ ] g
- ' ' . ' g'j '
, Q V;%. x 4 . _v. f z -
- w- s.g -
s
$ _ .- g :
+ =
1 e < w qt. l. . .
-$ )
{ fu
, ~g lW ? . l' !
.'%.-.. '.;'..__..,.' .. - . 5:. .. ' y
.'r:. A . ,$ ..
1 y y-
... p
.' " f- s
- C.
2
* (.'#
L .
. # g q -
g w - g . .
< - 5
. I'~ ,'. g' ' - (* M' { .. 4 .
$ :g, k d f5 .
~ Af .[ . A *[ .
y , ! S
~ ,o
=<
<. .u ; , -m ..
A
- e .'
..s .
E z ' l - u - ' 7. ; 1 .' $ 3 4 -
%,C-', ( , --
- p w
< y r...
}
c u
-. - y
. 7 s
g 1 -. , -
. fy. , ,. .
's -
_ Si$ s:f', i: ' t: 5 u
- rN
.. -' t l
k
, % ,l $
}_
%- r m .. -
.% vg <
E e 9g . m C g - a c 4.k .\ <
.. - 3
- f-kx F .
3g.; . . , I g 4 > 3 - .j- .- 6 '- e
. . c.. ,
. - D [ g.
{ j ! .' .d- - g ;. 'h- % ' .'- + r z m g w E w F. E Cw W
<<C 2 a$z l 33< *z *c j EC$z $0 E Ee 1 zz
<w-<
*e> - az <
5j< g[
<-4
ABSTRACT On November 21, 1985, Southern California Edison's San Onofre Nuclear Generat-ing Station, Unit 1, located south of San Clemente, California, experienced a partial loss of inplant ac electrical power while the plant was operating at 60 percent power. Following a manual reactor trip, the plant lost all inplant ac power for 4 minutes and experienced a severe incidence of water hammer in the feedwater system which caused a leak, damaged plant equipment, and chal-lenged the integrity of the plant's heat sink. The most significant aspect of , the event involved the failure of five safety-related check valves in the feed-water system whose failure occurred in less than a year, without detection, and jeopardized the integrity of safety systems. The event involved a number of equipment malfunctions, operator errors, and procedural deficiencies. This report documents the findings and conclusions of an NRC Incident Investigation Team sent to San Onofre by the NRC Executive Director for Operations in confor-mance with NRC's recently established Incident Investigation Program. W l l iii j
TABLE OF CONTENTS P,g[e Abstract............................................................. iii , The NRC Team for the San Onofre Event of November 21, 1985........... ix Acronyms and Abbreviations........................................... x 1. INTRODUCTION................................................... 1-1
- 2. DESCRIPTION OF TEAM ACTIVITIES................................. 2-1 2.1 General Approach........................................ 2-1 2.2 Interviews and Meetings................................. 2-1 2.3 Plant Data.............................................. . 2-2
- 3. NARRATIVE OF THE EVENT........ ................................ 3-1 3.1 Plant Status............................................ 3-1 3.2 El ectri cal Ground Faul t. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-2 3.3 Loss of Power........................................... 3-3 3.4 Fo u r-Mi n u te B l ac ko u t . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3- 6 3.5 Conflicting Requirements................................. ?-A 3.6 The Water Hammer........................................ 3-9 3.7 Steam Generators Boil Dry............................... 3-10
- 3. 8 Plant Cooldown.......................................... 3-11 3.9 Isolating the Leak...................................... 3-13
- 4. DESCRIPTION OF PLANT SYSTEMS................................... 4-1 4.1 General Design.......................................... 4-1 4.2 Main Steam System....................................... 4-1 4.3 S team Generator 810wdown Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . 4-2 4.4 Ma i n Fe e dwa te r Sy s tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
, 4.5 Condensate System....................................... 4-3 4.6 Auxi l ia ry Feedwater Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-4 4.7 Salt Water Cooling Systems.............................. 4-5 4.8 Turbine Plant Cooling Water Sys tem. . . . . . . . . . . . . . . . . . . . . . 4-6 4.9 Chemical and Volume Control System...................... 4-7 4.10 1 Residual Heat Removal System............................ 4-8 ' 4.11 Sa fe ty Inj ection Sys tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-8 4.12 Electrical Distribution System.......................... 4-9 4.12.1 General Description.............................. 4-9 i 4.12.2 220 KV System.................................... 4-10 1 4.12.3 4160-Volt System............. ................... 4-11 4.12.4 480-Volt System.................................. 4-13 l 4.12.5 120-Vo l t A C Sy s t em. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4- 14 v
TABLE OF CONTENTS (Continued)
.Pa_g 4.13 Safeguard Load Sequencing System.......................... 4-15 1 4.14 Diesel Generators......................................... 4-16 4.15 Containment Building Isolation............................ 4-17 4.16 Station Personnel......................................... 4-17 4.16.1 Administrative Organization...................... 4-17 -
4.16.2 On-Shift Organization............................ 4-18 4.16.3 Human Factors Affecting Operator Performance...................................... 4-19 4.17 Control Room Indication.................................. 4-21 . 4.17.1 Loss of Vital Bus 4.............................. 4-21 4.17.2 Loss of Utility Bus.............................. 4-22 4.17.3 Data Recording Systems........................... 4-23 4.18 NRC Emergency Response Organization....................... 4-24
- 5. LOSS OF POWER EVALUATION....................................... 5-1 5.1 Electrical System Operation. Leading to Auxiliary Transformer C Differential Relay Trip................... 5-1 5.1.1 Relay Trip...................................... 5-3 5.1. 2 Discussion...................................... 5-3 5.2 Uni t Trip and Loss of All AC Power. . . . . . . . . . . . . . . . . . . . . . 5-4
- 6. WATER HAMMER EVALUATION _ ............................. .. 6-1 6.1 Plant Conditions Leading to Water Hammer......... . 6-2 6.2 Water Hammer-Induced Damage....................... . 6-4 6.2.1 Piping and Piping Support Damage................ 6-4 6.2.2 Feedwater Loop 8 Flow Control Station Damage.................................. 6-5 6.2.3 AFW Piping Damage............................... 6-5 6.2.4 Valve Malfunctions and Damage................... 6-5 -
6.3 NRC Evaluations of Water Hammer......................... 6-7 6.3.1 History and Focus............................... 6-7 6.3.2 SONGS-1 Water Hammer History and Evaluations.... 6-9 I 6.4 Valve Inservice Testing................................. 6-9 i 6.5 Feedwater System Check Valve Maintenance................ 6-12 6.6 Feedwater Train Noise Investigation..................... 6-12 6.7 Valve Failure-Related Findings.......................... 6-13 l l l l l l v1 l l l .- . - - _ _ - _ _.
TABLE OF CONTENTS (Continued) Page
- 7. HUMAN FACTORS EVALUATION....................................... 7-1 7.1 Introduction............................................ 7-1 7.2 SONGS Personnel Performance............................. 7-1 7.2.1 Operator Performance............................. 7-1 7.2.2 Other Si te Personnel Performance. . . . . . . . . . . . . . . . . 7-8
- 7. 3 Procedures.............................................. 7-11 0 7.3.1 ENS Communications Problems...................... 7-11 7.3.2 NRC Incident Response Plan Implementation........ 7-15
- 8. OTHER EQUIPMENT AND SYSTEM EVALUATIONS......................... 8-1 8.1 Auxiliary Trans former C Secondary Side. . . . . . . . . . . . . . . . . . . . 8-1 8.2 Safety Injection Annunciator.............................. 8-1 8.3 SLSS Remote Surveillance Panels........................... 8-2 8.4 Flash Evaporator Unit..................................... 8-2 8.5 Turbine Breakable Diaphragms (Rupture Disks).............. 8-2 8.6 Reactor Coolant Pump B Thrust Bearing Temperature Indication.................................... 8-3 8.7 Steam Generator Blowdown Isolation........................ 8-3 8.8 RHR Va l ve I n te rl o c k. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8-3 8.9 Event Recording Systems................................... 8-3 8.10 Emergency Notification System............................. 8-4 8.11 Safeguards System......................................... 8-4
- 9. PRINCIPAL FINDINGS AND CONCLUSIONS............................. 9-1 APPENDIXES A Memorandum from W. J. Dircks, Executive Director for Operations, to the Commission, " Investigation on November 21, 1985 Event at San Onofre Unit 1 Will Be Conducted by an Incident Investigation Team (IIT),"
. November 2 2 , 19 8 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A - 1 B Plant Conditions When Water-Hammer Occurred and Estimated Piping Support Loads.................................. B-1 C Regulatory Review of Potential for Water Hammer a t S a n O n o f r e U n i t 1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C- 1 i
vii
THE NRC TEAM FOR THE SAN ONOFRE EVENT OF NOVEMBER 21, 1985 Thomas T. Martin, Team Leader Matthew Chiramal William G. Kennedy Wayne D. Lanning Aleck W. Serkiz Steven K. Showe TEAM SUPPORT STAFF Walter E. Oliu, Division of Technical Information and Document Control iX
ACRONYMS AND ABBREVIATIONS ac Alternating Current ACO Assistant Control Operator AFW Auxiliary Feedwater ARMS Area Radiation Monitoring System ASB Auxiliary Systems Branch (NRC) ASME American Society of Mechanical Engineers ATWS Anticipated Transient Without Scram BWR Boiling Water Reactor - CB Circuit Breaker CCW Component Cooling Water C0 Control Operator . CRS Control Room Supervisor CV Control Valve CVCS Chemical and Volume and Control System dc Direct Current EAL Emergency Action Level ENS Emergency Notification System E0 Emergency Officer E0I Emergency Operating Instruction EPIP Emergency Plan Implementation Procedure ERMS Effluent Radiation Monitoring System ESF Emergency Safeguards Features FCV Flow Control Valve FEMA Federal Emergency Management Agency FI Flow Indicators FW Feedwater FWS Feedwater Support GDC General Design Criteria gpm gallons per minute HQDO Headquarters 6uty Officer (NRC) HVAC Heating, Ventilation, and Air Conditioning HX Heat Exchanger IIT Incident Investigation Team IRC Incident Response Center IST Inservice Testing kV Kilovolts kVA Kilovolts Ampere kW Kilowatts - lbf Pounds Force LCV Letdown Control Valve LCV Level Control Valve . LER Licensee Event Report LMFW Loss of Main Feedwater LOB Loss of Bus LOCA Loss of Coolant Accident LOP Loss of Offsite Power LP Low Pressure LR Level Recorder mA Mf111 amperes MCC Motor Control Center MFW Main Feedwater M0 Maintenance Order x
MOD Motor-0perated Disconnect MOV Motor-Operated Valve MVA Megavolt Amperes MWe Megawatt (Electric) MWt Megawatt (Thermal) NPE0 Nuclear Plant Equipment Operator NRC Nuclear Regulatory Commission NRR Office of Nuclear Reactor Regulation (NRC) NSSS Nuclear Steam Supply System OA/FA Oil-Air / Forced Air 01 Operating Instruction
. ORMS Operational Radiation Monitoring System PORV Power Operated Relief Valves POV Power-0perated Valve ppb parts per billion npm parts per million l PR Pressure Recorder PT Potential Transformer l PWR Pressurized Water Reactor PZR Pressurizer RCS Reactor Ccolant System R00 Regional Duty Officer RHR Residual Heat Removal R0 Reactor Operator RPF0 Reactor Plant First Out (Annunciators)
RWST Refueling Water Storage Tank SCE Southern California Edison Company SDG&E San Diego Gas and Electric Company SEB Sphere Enclosure Building SER Safety Evaluation Report SG Steam Generator SGWH Steam Generator Water Hammer SIS Safety Injection System SLSS Safeguard Load Sequencing System SONGS-1 San Onofre Nuclear Generating Station, Unit 1 SRI Senior Resident Inspector SR0 Senior Reactor Operator SSAM Shift Supervisor's Accelerated Maintenance (Order) SST Station Service Transformer STA Shift Technical Advisor T Average Temperature ave TBCW Turbine Building Cooling Water TI Temperature Indicators TIC Temperature Indicating Control TPCW Turbine Plant Cooling Water T Reference Temperature ref TR Temperature Recorder USI Unresolved Safety Issue V Volts VAC ac voltage VCT Volume Control Tank VDC dc voltage WOG Westinghouse Owner's Group YR Steam Flow / Feed Flow / Level Recorder xi
1 INTRODUCTION The San Onofre Nuclear Generating Station, Unit 1, operated by the Southern California Edison Company (SCE), is a 450 MWe Westinghouse pressurized water reactor located on the Pacific Ocean, approximately four miles south of
, San Clemente, California. The plant received an NRC operating license in 1967.
At 4:51 a.m., on November 21, 1985, the plant was operating at 60 percent power, when a ground fault was detected by protective relays associated with a trans-former which was supplying power to one of two safety-related 4160V electrical buses. The resulting isolation of the transformer caused the safety related bus to de-energize, which tripped all feedwater and condensate pumps on the east side of the plant. The pumps on the west side of the plant were unaffected since their power was supplied from another bus. The continued operation of the west feedwater and condensate pumps, in combination with the failure of the east feedwater pump discharge check valve to close, resulted in over pressurization and rupture of an east side flash evaporator low pressure heater unit. The operators, as required by emergency procedures dealing with electrical systems, tripped the reactor and turbine generator. As a result, the plant experienced its first complete loss of steam generator feedwater and inplant ac electrical power since it began operation. The subsequent 4-minute loss of inplant electric power started the emergency diesel generators (which by design did not load), de-energized all safety-related pumps and motors, significantly reduced the number of control room instrument indications available for operators to diagnose plant conditions, produced spurious indications of safety injection system actuation, and caused the NRC red phone on the operator's desk to ring. Restoration of inplant electric power was delayed by an unexpected response of an automatic sequence that should have established conditions for delayed remote manual access to offsite power still available in the switchyard.
. The loss of steam generator feedwater was the direct result of the loss of power to the two main feedwater and one auxiliary feedwater pump motors, and the designed 3 minute startup delay of the steam powered auxiliary feedwater , pump. The loss of the feedwater pumps, in combination with the failure of four additional feedwater chech valves to close, allowed the loss of inventory from all three steam generators and the partial voiding of the long horizontal runs of feedwater piping within the containment building. The subsequent automatic start of feedwater injection by the steam powered auxiliary feedwater pump did not result in the recovery of steam generator level because the backflow of steam and water to the leak in the evaporator carried the auxiliary feodwater with it. Later, operators isolated the feedwater lines from the steam genera-tors, as required by procedure, unknowingly initiating the process of refilling the feedwater lines in the cantainment building. Before all feedwater lines were refilled, a severe wate.r hammer occurred that bent and cracked one 1-1
l feedwater pipe in the containment building, damaged its associated pipe supports and snubbers, broke a feedwater control valve actuator yoke, stretched the studs, lifted the bonnet, and blew the gasket from a 4-inch feedwater check valve. The damaged check valve developed a significant steam-water leak, the second leak in the event. The second leak, in combination with an earlier inadvertent re-establishment of steam generator blowdown, caused all three steam generator water levels to drop below indicating levels. Steam from all three steam generators fed the leak, because of the absence of individual main steam isolation valves. Despite these problems, operators later succeeded in recovering level indication - in the two steam generators not directly associated with the feedwater piping leak. With the reestablishment of steam generator levels, the operators safely brought the plant to a stable cold shutdown condition, without a significant release of radioactivity to the environment (the pre-existing primary to second-ary leak was not exacerbated) and without significant additional damage to plant equipment. On the day following the event, and in conformance with the recently established Incident Investigation Program, the NRC Executive Director for Operations sent an NRC Team of technical experts to the site. (For the directive establishing the Team, see Appendix A.) The original five-member Team, augmented by an addi-tional staff member, was selected because of its broad experience in operating plant event analyses, with individual Team members having specific knowledge and experience in operations, human factors, electrical and reactor systems, and water hammer phenomena. The Team was directed to (1) determine what happened; i (2) identify the probable causes; and (3) make appropriate findings and con-clusions to form the basis for possible follow-on actions. This report docu-ments the results of the Team's efforts in identifying the circumstances and causes of the event, together with its findings and conclusions. The scope of this fact-finding effort was limited to the circumstances surround-ing the events of November 21, 1985 including operator and NRC actions, equip-ment damage and malfunctions, equipment maintenance and testing history, and regulatory involvement. l Section 2 describes the methodology used by the Team to collect and evaluate l information about the event. Section 3 provides a narrative and detailed sequence of events, reconstructed from analysis of operator and NRC interviews ' and logs, event recorders, and system descriptions. Data to confirm operator observations were sketchy, at best, because the loss of station power inter- - rupted operations of crucial recording instruments. Section 4 provides a summary description of how San Onofre, Unit I mechanical , I and electrical systems involved in this event function and interact. Under-standing the major differences between this plant and more recently designed pressurized water reactors will clarify the basis for operator actions. Section 5 discusses the performance of plant electrical equipment, the mal-function of which initiated tFe sequence of events and complicated the operator response to it. Sections 5 through 8 relied heavily on available results from Southern California Edison's analysis and troubleshooting activities of equip-l ment involved in the event. 1-2 I
Section 6 and its associated appendices discuss the results of the Team's review of the development of conditions that resulted in the water hammer, and the resulting damage it caused. This section also examines the maintenance and testing history of the check valves which failed, and the communications between NRC and SCE on (1) the inservice testing program for safety-related check valves, (2) the design of the auxiliary feedwater system, and (3) the prevention or mitigation of water hammer effects. Section 7 discusses the performance of operators and NRC staff dur ing the event and the human factors considerations which affected them. Section 8 discusses additional noteworthy equipment and system problems that occurred during the event. , Finally, Section 9 presents the Team's principal findings and conclusions, based on information available to the Team at the time the report was compiled. It should be noted that the root causes of selected equipment failures are still under investigation. However, with the possible exception of the identifica-tion of the root cause of the feedwater system check valve failures, it is unlikely that new significant information relative to what happened and why will be developed. 1-3
l l I 2 DESCRIPTION OF TEAM ACTIVITIES 2.1 General Approach In general, the investigative methods used by the San Onofre Incident Investigation Team were based on the experience and methodology developed by the Incident Investigation Team for the Davis-Besse event of June 9,1985. To assure continuity and consistency in Team activities, one member from the Davis-Besse Team served on the San Onofre Team. The Team collected and evaluated a variety of information to determine the sequence of operator, plant, and equipment responses during the event and the causes of equipment malfunctions and operator errors. The sequence of responses was difficult to reconstruct because digital data from the Technical Support Center computer was unavailable. Instead, the sequence was developed primarily from interviews with personnel involved in the event. The Team inspected the equipment which malfunctioned, the equipment and piping damaged by water hammer, and the control room instrumentation and controls. The Team also interviewed plant management and NRC Region V personnel at the site about their knowledge of plant response and operator activities. Personnel at Southern California Edison Corporate headquarters and NRC headquarters were also interviewed to explore the design and regulatory history of the auxiliary feedwater system (as it related to water hammers) and the inservice testing program for Unit 1. The equipment which malfunctioned or contributed to the water hammer event was quarantined soon after the event so that troubleshooting and examinations could be performed systematically, and so that evidence concerning the root cause of failures would not be lost or destroyed. The root causes, which have yet to be definitively established in only a few cases, are being determined by Southern California Edison (SCE) personnel and equipment vendors using procedures agreed upon by the Team. 2.2 Interviews and Meetings The event occurred the week before Thanksgiving and about a week before a major planned plant outage, disrupting SCE staff plans for work and vacations. The prompt initiation of the investigation by the Team upon its arrival the weekend before the holiday, and the magnitude of the overall troubleshooting and in-vestigative effort, tended to overwhelm the available onsite technical staff. To the SCE staff's credit, they were always cooperative and open with their ideas and information. The formal investigation began on the morning of November 23 when the Team met with SCE personnel to obtain an overview of their understanding of the event. Following this meeting, the Team began interviews with operating person-nel. The Team placed a high priority on interviewing personnel on duty at the time of the event to learn about the actions they took and the observations they made. The Team recognized that the quicker these interviews could be held, the more information those being interviewed would remember. 2-1
The Team split into two groups to accelerate the completion of the interviews. Operator interviews were completed in two days. All interviews and some meetings were recorded by stenographers who prepared and typed transcripts and made them available to the Team and interviewees on the following day. Those interviewed had the opportunity to review the transcripts and complete errata sheets when necessary. Interviews were, in general, scheduled with personnel in decreasing order of their seniority within the shift, beginning with the Shift Superintendent and proceeding to those less senior. The rationale for this sequence was to move ' from general to specific information. Thus, the Team obtained information on overall plant operations before obtaining information on the detailed actions - of specific operators. Some personnel were interviewed more than once when the Team needed additional - clarifying information. Table 2.1 contains a listing of the 51 interviews and meetings the Team conducted. 2.3 Plant Data Because of the nature of this event, the plant data available to evaluate this event was significantly deficient in comparison to information normally available after a reactor trip. No precise digital data were available before or during the event. The only data permanently recorded for post-trip evaluation included:
- 1. Strip charts from the trend recorders *
- 2. Oscillograph trace recordings
- 3. The strip chart from the control room event recorder
- 4. Logs maintained by operators, emergency coordinators, emergency services, and security personnel.
The Foxboro computer, which is part of the Technical Support Center and normally provides accurate time recordings of plant conditions, was not available because of the troubleshooting activities in progress before the reactor trip. A description of this data collection equipment is found in Section 4.17.3. SCE provided the Team with numerous color photographs documenting plant layout and the as-found condition of equipment. These photographs, some of which are included in this report, contributed significantly to the Team's understanding and evaluation of the event. The Team also examined over 600 documents and - other pieces of evidence during its investigation. All the information used by the Team has been catalogued and will be placed in the local and NRC Public Document Rooms in a special San Onofre file. .
- The majority of these were inoperable during the 4-minute loss of ac power.
2-2
Table 2.1 Interviews and Meetings Conducted by the San Onofre Team Date Time Meeting / Interview 11/23/85 10:15 Entrance Meeting on Incident Investigation 11/23/85 13:15 Interview of Electrician 11/23/85 14:00 Interview of Test Technician 11/24/85 08:55 Interview of Supervisor of Coordination, Acting Unit Superintendent 11/24/85 08:57 Interview of Shift Supervisor, Midnight Shift 11/24/85 10:20 Interview of Control Operator 11/24/85 11:30 Interview of Control Room Supervisor 11/24/85 12:20 Interview of Shift Technical Advisor 11/24/85 13:35 Interview of Nuclear Plant Equipment Operator 11/24/85 14:00 Interview of Nuclear Plant Equipment Operator 11/24/85 14:20 Interview of Assistant Control Operator, Control Room Operator (trainee) 11/24/85 15:10 Interview of Assistant Control Operator 11/24/85 15:50 Interview of Nuclear Plant Equipment Operator 11/25/85 09:25 Interview of NRC Senior Resident Inspector 11/25/85 09:30 Interview of Shift Superintendent, Day Shift 11/25/85 10:15 Interview of Control Room Supervisor, Day Shif t
, 11/25/85 10:25 Interview of Control Room Operator, Day Shift 11/25/85 14:15 Interview of Supervisor, Fire Protection Services 11/25/85 15:00 Manager of Safety Emergency Preparedness 11/25/85 15:40 Interview of NRC Resident Inspector
- Transcripts were made of all meetings and interviews listed.
2-3
Table 2.1 Interviews and Meetings Conducted by the San Onofre Team (continued) 11/25/85 16:55 Interview of NRC Resident Inspector 11/25/85 18:10 Interview of Fire Captain, San Onofre Fire Department, Shift Captain 11/26/85 10:30 Interview of NRC Resident Inspector - 11/26/85 13:00 Water Hammer Conference 11/26/85 16:00 Interview of Emergency Medical Technician 11/26/85 16:30 Interview of Fire Watch Rover 11/26/85 18:10 Interview of Assistant Control Operator, Control Room Operator (trainee) 11/27/85 08:15 Interview of Fire Watch Rover 11/27/85 10:05 Interview of Station Manager 11/27/85 11:18 Meeting of SONGS Security, Station Security Manager, Computer Supervisor, Computer Engineer (Safeguards Information) 11/27/85 14:00 Interview of Chemistry Supervisor 12/11/85 09:00 Interview of Manager of Station Operations 12/11/85 10:00 Interview of Mechanical General Foreman 12/11/85 11:07 Interview of Shift Superintendent 12/12/85 08:10 Interview of Plant Superintendent, Unit 1 12/12/85 09:55 Interview of Shift Superintendent 12/12/85 11:00 Interview of Health Physics Technicians 12/12/85 11:30 Interview of Health Physics Technician . 12/12/85 13:20 Interview of Nuclear Plant Equipment Operator 12/12/85 16:40 Interview of Health Physics Supervisor 12/12/85 17:25 Interview of Health Physics Technician 12/13/85 07:45 Interview of Control Operator 2-4
l Table 2.1 Interviews and Meetings Conducted by the San Onofre Team (continued) 12/13/85 08:40 Interview of Nuclear Operations Assistant 12/13/85 13:00 Transcript of Proceedings in Rosemead, California 12/18/85 10:10 Interview of NRC Task Manager 12/18/85 11:20 Interview of Former NRC Auxiliary Systems Branch Engineer 12/18/85 13:00 Interview of NRC Duty Officer 12/18/85 15:05 laterview of NRC Systematic Evaluation
- Program Manager 12/20/85 10
- 10 Interview of NRC Engineer 12/20/85 11:46 Interview of Senior Reactor Events Analyst
, 12/26/85 13:25 Interview of Assistant Director for Operating j Reactors i 12/26/85 15:00 Interview of Assistant Director for Safety Assessment
- - 12/27/85 09
- 00 Interview of NRC Engineer, Engineering Issues Branch 12/27/85 10:28 Interview of Acting Branch Chief, Mechanical Engineering Branch 12/27/85 15:25 Interview of Deputy Director, Division of Human
- Facters Technology
, 12/27/85 15:48 Interview of Assistant Director for Generic Issues i
2-5 4
3 NARRATIVE OF THE EVENT This section provides a step-by-step description of events in the early morning hours of November 21, 1985, at San Onofre, Unit 1, that led to a loss of inplant power and a severe incidence of water hammer. The detailed record of events normally provided by the plant computer was not available because the computer ~ had not been reset following earlier electrical ground troubleshooting activi-ties. The NRC Team based this narrative and the sequence of events listed in Table 3.1 on a composite of operator and management interviews, operator logs, readings from the plant's oscillograph trace recorder, and interpretations of strip charts from plant trend recorders. The interviews indicate that various plant personnel heard a series of unex-plained loud " bangs" and muffled " booms" during the event. In addition, secur-ity personnel saw flashes of light above a transformer in the site's restricted area. To the extent possible, the sights and sounds heard by the plant staff have been correlated with the response of plant parameters recorded by the instruments previously mentioned. 3.1 Plant Status On November 20,1985 at 11:30 p.m. ,* the midnight shift of operators assumed control of the San Onofre Nuclear Generating Station Unit 1. During the day shift, the generator load had been decreased from 407 MWe to 250 MWe (licensed power is 450 MWe) because of a tube leak in one part of the main condenser. Saltwater from the Pacific Ocean is used to cool the condenser and a small amount of it was leaking into the condenser. One of the circulating water pumps was stopped so that the leak could be fixed. With only one-half of the condenser tvailable to condense steam, the generator load (and steam flow rate) had been reduced accordingly. The saltwater leak into the condenser contaminated the feedwater with corrosive chlorides, which would adversely affect the steam generator tubes. To reduce the concentration of chlorides in the steam generator, the rate of secondary blowdown had been increased from about 40 to about 100 gallons per minute for each steam generator. (Blowdown is the water being removed from the steam generators.) The blowdown flow rate was unusually high, and will adversely affect steam generator level recovery later because it was equal to about two-thirds of the total flow to each steam generator that could be provided by the auxiliary feedwater system. Except for saltwater leakage into the condenser, no other conditions were affect-ing plant operations. However, both pressurizer power operated relief valves (PORVs) were isolated (a condition permitted by the plant's Technical Specifi-cations) because one valve was leaking and the block valve for the other PORV had failed a surveillance test. No other ongoing tests or changes to the plant status were planned. The oncoming shift of five control room operators (2 SR0s
*All times represent Pacific Standard Time unless otherwise specified.
3-1
and 3 R0s) and two plant equipment operators had barely completed shift turnover procedures when, at 10 minutes before midnight, an alarm sounded in the control room indicating that a 100 percent ground was detected by the ground detector on 4160-volt bus IC. (An electrical ground is an undesirable path for current from the power source to some other conducting body.) 3.2 Electrical Ground Fault Bus 1C is a 4160-Volt safety-related bus through which electrical power flows to safety-related equipment (e.g. , the feedwater/ safety injection pump, the - charging pump, the auxiliary feedvater pump) and other equipment during normal plant operation. Power is normaily supplied to bus 1C from offsite, through - the station switchyard and, in an emergency, can be supplied from one of two I backup diesel generators. The 4160-Volt bus 2C is the backup for bus 1C in that it powers the redundant train, i.e. , other side, of safety-related equip-ment. Even though an electrical ground was present on bus IC, such a condition does not interrupt power to the equipment and thus the operation of plant equip-ment was routine. The alarm did not identify the precise location of the ground. It could have existed anywhere from the transformer secondary windings to the motor of a safety-related pump or other loads connected to bus IC. The development of a second ground, however, could result in an electrical fault and in protective relay actuation and isolation of the faulted circuit (e.g. , loss of power to the bus); or )f the ground was in a motor, it could damage it. Consequently, after midnight, two electrical technicians were called to the site to assist in troubleshooting the ground. After the alarm annunciated indicating the ground on bus IC, the reactor oper-ators referred to a written ground fault procedure for locating and isolating the ground. For several hours, the operators swapped, started and stopped equip-ment according to the procedure in an attempt to locate the ground. The Super-visor of Coordination, a member of Unit 1 management (the Acting Plant Superin-tendent), who is a licensed SRO, was called by the Shift Superintendent and came to the site at about 3:00 a.m. to provide assistance. The Shift Superintendent thought the situation would require de-energizing bus IC, a decision that would affect power operation and the Technical Specifications which govern plant oper-ations. (The Technical Specifications require that the plant be shut down if bus 1C is de-energized for a period of 8 hours or more.) After the Supervisor of Coordination arrived, the actions taken and the remain-ing options available to isolate the ground were discussed. Operators had com- - plied with the ground fault procedure to the extent of having tested all equip-ment connected to bus IC, except for the west main feedwater pump. In order to test the pump, which required that it be removed from service, the Unit 1 elec- . trical load had to be reduced to ensure continued operation without a reactor trip. Accordingly, preparations were made to stop the feedwater pump by reduc-ing the unit load from 250 to 150 MWe and realigning support systems. In the meantime, the two electrical test technicians had arrived onsite. After verifying that the ground was not in any load connected to the bus (except for the west feedwater pump), the operators followed the ground fault procedure, until the step that required that bus 1C be de-energized. However, at this 3-2
point the Shift Superintendent, with suggestions from the electrical test tech-nicians and with his management's concurrence, improvised a troubleshooting method not included in the procedure. The proposed method of troubleshooting was seen as having an advantage in that bus IC need not be de energized. Instead of de-energizing bus IC, the improvised troubleshooting method involved the operators connecting it in parallel with bus IA, and disconnecting its nor-mal power supply (transformer C). This paralleling operation was performed three times. When the power supply from transformer C to bus 1C was disconnected, the ground alarm cleared, which indicated to the operators that the ground was on the secondary side of transformer C feeding bus IC. This finding occurred , at about 3:30 a.m., and eliminated the need to remove the west main feedwater pump from the bus since the fault was known to be on the transformer side of the bus. Subsequently at 4:30 a.m., with bus 1C aligned to receive power from bus 1A rather than transformer C, the unit load was increased from 150 MWe to 250 MWe. Transformer C with the ground fault still present was left energized and supplying bus 2C. This unusual switchgear alignment is not covered in plant procedures. Bus 1C had not been de-energized because the operators knew that this would invoke a Technical Specifications Action Statement, a condition that they believed should always be avoided. This situation was discussed several times with the Supervisor of Coordination. Plant Technical Specifications require that buses 1C and 2C always be energized (except during refueling); if bus 1C is de-energized, the Action Statement requires that it be returned to operable status within 8 hours or that the plant be shutdown. The two electrical technicians were dispatched to visually inspect transformer C and its switchgear in an attempt to locate the ground. It was while they were inspecting the switchgear that they heard a loud " boom". At the same time a security guard saw a flash of light from the vicinity of the top of the trans-former near the technicians. It was 4:51 a.m. 3.3 Loss of Power In the control room, operators also heard the " boom" while the background noise from rotating machinery wound down amidst the sounds of numerous alarra and the illumination from rows of annunciator lights. Immediately, all five operators surveyed the alarm panels and control boards to diagnose the situation. The Shift Superintendent noticed that transformer C had been isolated by differen-tial relay protection, which resulted in a loss of power to bus 2C. From among . the multiple, nonassociated alarms displayed on the panels, he identified the transformer C differential relay alarm on the turbine first-out panel and quickly determined that vital bus 4 had been de-energized. He directed operators to , trip (shut down) the reactor pursuant to the immediate actions specified in the procedure for loss of this vital bus. Two operators pushed each of the two manual reactor trip buttons (only one of which is required to trip the reactor) and another operator then pushed the unit trip button. Within 20 seconds the operators had identified the initiator of the event, and had manually tripped the reactor and turbine generator in accordance with the plant procedure for loss of the vital bus. At that moment, only emergency lights illuminated the control room as normal station lighting went black when inplant ac power was lost. 3-3
Immediately following the reactor trip, the red telephone, part of the emergency notification system connected directly to NRC headquarters in Bethesda, Maryland, rang in the control room. Although pre occupied with examining the control boards and alarm panels, and while watching the reactor operators respond to the reactor trip, the Shift Superintendent finally answered the telephone. The voice on the telephone was that of the NRC Duty Officar in Bethesda (evidently the phone had rang at both ends). The Shift Superintendent was puzzled as to why NRC wanted information about or could possibly have known that a reactor trip had occurred less than one-half minute before. The exchange of <luestions and answers that ensued between the Unit I control room and the NRC Duty Officer was a mixture of miscues and incomplete communication. Finally, the Shift Superintendent implied to tr< Duty Officer that he was too busy to talk, told him - that he would call him back, e f *.ohf him the unit was stable. The NRC Outy Officer did not ask the cug rot w.n a g ators about which emergency class was declared. Evidently, when bus IC icst powe*, the m rgency notification system responded to the power interruption by signaling W red telephone in the control room and at the NRC Operations Center to rf% p ch party thought that the other party had called, and as a consequence, meanOgful dialogue was not established. During a second telephone call, the Shift Superintendent indicated that an Alert would probably be declared, but subsequent evaluation led to the declaration of an Unusual Event pursuant to the site's emergency plan. However, the event was not reported to the NRC as an Unusual Event, although the control room operators and site personnel knew that this emergency action level had been declared. As a result of the mixet corcunicat M s, the NRC Duty Officer believed that an Alert had been decle W . Lubsequi# 1y, the NRC requested that an open, continu-ous telephone line bt Wint >ined to the control room, an arrangement which is normally reserved fee an A %rt declaration. The Shift Superintendent did not understand the need toe w.h a request, particularly since it distracted him from his responsibilities in the control room. He then requested the Super-visor cf Co ordination to communicate with t% NRC. Later, when the NRC Resident Inspector arrived in the cor. trol room at about 6:00 a.m., he was requested to handle the open telephene lint with the NRC. If the plant had been in a normal switchgear alignment, isolating transformer C would have de-energized buses 1C and 2C. But because of the abnormal alignment, the isolation of transformer C de energized only bus 2C, because bus 1C was powered by the main turbine generator via bus 1A. Thus, all inplant power was not instantaneously interrupted. About one-half of the equipment lost power when bus 2C (i.e., transformer C) was lost, and 20 seconds later the remaining - equipment lost power when the unit was manually tripped. As a result, the west feedwater pump (on bus IC) continued to operate after the east feedwater pump (on bus 2C) lost power. . At about 4:50 am, in the 4kV switchgear room located below the turbine building mezzanine level, a roving fire watch had come to relieve the fire watch sta-tioned in the room. Shortly thereaf ter, they heard a cannon-like sound and felt the floor vibrate. One of the fire watches described the noise as a "muf-fled howitzer." The test technicians outdoors described it as a loud " boom." During the 20-sacond interval that the west main feedwater pump continued to operate, the single 12-inch diameter check valve at the discharge of the east 3-4
feedwater pump stuck open. As a result, the high pressure (1300 psig) discharge water from the west pump flowed backwards through the east feedwater pump and overpressurized the east condensate piping and components (design pressure is only 350 p3ig). The low pressure flash evaporator condenser had been overpressurized by the backflow of feedwater from the west feedwater pump. The feedwater at high tem-perature and pressure ruptured a tube in the flash evaporator condenser and ballooned the rectangular shell of the flash evaporator until a 20-foot long 2-foot-wide fishmouth split occurred along the welded seam and relieved the pressure. Water and steam covered the east part of the turbine building and . activated fire alarms several levels below the turbine deck. Post event investigation revealed that a power supply cable from transformer C to bus 1C had failed. The cable had shorted and caused two large openings in the metal jacket covering the wires. When the cable failed, a loud noise, per-haps an electrical arc, occurred. Thus, the noises heard by the fire watches and test technicians may have been the rupture of the flash evaporator located on the turbine building deck above the 4kV room, or the failure of a cable in a cable tray located in the turbine building, or some combination of the two. As the overhead lights went out, two alarm lights appeared on the fire protec-tion annunicator panel located in the 4kV room and started blinking, an indica-tion that there wa.s a fire. One of the alarms was for the hydrogen seal area and the other was for the lube oil storage area. One fire watch carefully went out into the darkness with his flashifght to investigate the noise and alarms. He instructed the other fire watch to report the fire alarms to the control room by telephone. Several attempts by the fire watch in the 4kV room to call the control room failed because both numbers on his call sheet were busy. Finally, he called the site fire department, assuming that there was a fire. The fire-watch officer at the site fire department asked him to go outside to see if there was a fire. ' When he opened the door he saw what he thought was smoke and the silhouettes of operators running around. He reported his observations to the officer, who then instructed him to go back and ask one of the operators where the fire was located. When he reported back to the officer that the operators were too busy to talk to him, the officer indicated he would dispatch the fire brigade. The fire apparatus, a Mack 50-foot telesquirt, triple-combination pumper, arrived at the turbine building within 3 minutes. Instead of a fire and smoke,
- they found only steam. They also discovered that the foam deluge system had activated around the lube oil storage tank due to steam escaping from the traps on the feedwater pump shaft seal. They remained on the scene to support the op.erators in their recovery activities.
When the operators pushed the unit trip button, all inplant ac power was lost. Consequently, the west feedwater pump stopped and may have reversed its rota-tional direction because its discharge check valve also stuck open and did not prevent backleakage from the steam generators. Reverse rotation could have camaged the pump. (Note - The feedwater pumps are also the safety injection 3-5
pumps.) The trend recorder data show that the potential existed for both trains of the condensate system to be overpressurized by the backflow from the steam generators. (Post-event examinations revealed that all five check valves be-tween the pumps and the steam generators failed to perform their safety function of preventing backleakage.) For this event, the failed tube in the flash evap-orator condenser probably relieved the pressure surges and prevented further overpressurization. 3.4 Four-Minute Blackout To an observer on the cliff above the beach overlooking the site at 4:51 a.m. , ' Unit I would have appeared mysteriously dark and the site abnormally silent. - (Units 2 and 3 were both shutdown for refueling.) When bus 2C lost power from transformer C, emergency diesel generator 2 started and achievcd rated speed. Emergency diesel generator 1 started when bus 1C de energized after the main - generator was tripped. According to the design for the onsite emergency power
, supply, the emergency diesels do not automatically provide power to emergency equipment af ter a loss of offsite power or de energization of buses 1C and 2C.
Thus, until operators manually load the diesel generators or restore offsite power to the buses, San Onofre Unit 1 is designed to be without any inplant ac power. Only the plant dc system and the inverter-supplied (battery powered) I vital buses remain operational. Without any ac power, the normal lighting in the control room and throughout the plant is lost. I After the control room operators manually tripped the plant, they immediately completed the necessary actions from memory. The Control Room Supervisor then read aloud the appropriate steps from the Emergency Operating Instructions (E0I) for a reactor trip or safety injection. The third step of the procedure requires operators to verify that buses 1C and 2C are energized. The operators found that they were de-energized. The E01 being read from directed the opera-tors to the E01 for loss of all ac power. The second step in the latter proced-ure requires the operator to determine if there is an indication of a safety injection (SI) actuation on the annunciator window located on the first out alarm panel. The alarm window and other instrumentation indicated that there was an SI actuation. However, the SI alarm was spurious. Because the operators took appropriate steps to verify that SI had not occurred, the spurious nature of the SI alarm was quickly determined. The next step in the E0I was to verify that the automatic sequencer had operated upon loss of voltage. The sequencer automatically realigns circuit breakers such that only an operator action to close a single breaker is necessary to restore inplant ac power via the main station transformer and offsite power. - However, the light that would indicate that the sequence was completed and that the final breaker should be closed to restore power did not illuminate. The operators waited for more than 2 minutes for the sequence to be completed and . then assumed the sequencer had failed. They then took manual action to actuate the necessary circuit breakers. At about 4:54 a.m. , the operators first attempted to complete the sequence to restore offsite power to buses 1C and 2C. All control room personnel watched the operator on the electrical board fail four times to close the electrical breakers necessary to restore power to the station. Finally, on the fifth attempt, and 4 minutes af ter power was lost, the buses were eneroized and the station blackout was over. 3-6
The blackout could have been longer if operator attempts to restore offsite power had not been successful. The E0I for loss of ac power lacks criteria for how long the plant can be without power before the diesel generators are to be loaded. In this case, the operators as required by procedures restored power from the switchyard rather than connecting the buses to the emergency diesel generators. (Had priority been given in the E0I to restoring power using the emergency diesel generators, power could have been restored in less than a min-ute.) In this 4-minute period, the feedwater lines started to empty, and steam pockets were formed in the feedwater piping, because of the five faulty check valves. The loss of the vital bus caused by the loss of transformer C caused multiple alarms on the annunicator systems and loss of indication and trend recorders in the control room. Remembering the large number of annunciator lights on the , panels, an operator indicated in an interview that he was shocked to see that the reactor had not tripped (with normal power distribution, the loss of trans-former C results in a reactor trip automatically). With few exceptions, all the trend re. corders failed and the recorded data for critical system parameters were lost during the blackout. Although these instruments are normally used by the operators to control the plant, adequate information from other instruments was available to the operators to understand the event and maintain plant safety. However, trends of critical parameters during the blackout phase were not avail-able to guide operator actions. Fortunately, the blackout lasted only 4 minutes and no additional complications occurred. The clock in the control room stopped for the 4 minutes while the power was lost. Subsequently, this clock was used to identify times when log entries for subsequent operator actions were made. Accordingly, the times entered in the logs and given by the operators in interviews had to be adjusted. After the reactor trip, the water levels in the three steam generators dropped below the setpoint for actuating the auxiliary feedwater system. The turbine-driven auxiliary feedwater pump received a start signal within seconds after the reactor trip. Although an actuation signal existed, the motor-driven pump does not receive a start signal until power is available. Because safety bus 1C was de-energized, the motor-driven pump could not start. The turbine-driven auxiliary feedwater pump started and began its warmup cycle. = Unlike turbine-driven auxiliary feedwater pumps at most other plants, this pump is designed to increase speed gradually for about 3 minutes before it delivers water to the steam generators. (This warmup period minimizes the potential for overspeed trips inherent in turbine-driven pumps when accelerated rapidly to full speed.) However, until the warmup cycle was ccmpleted, there was a total loss of feedwater to the steam generators. Af ter power was restored, the E01 for loss of ac power returned the operators to the E01 for reactor trip or safety injection. Following the E0I for reactor trip response, the operators closed the atmospheric steam dumps, thereby stop-ping a brief steam relief from the steam generators. At this time flow from the auxiliary feedwater system was verified to be about 135-150 gpm to each of the steam generators. The E01 then required that the main feedwater flow path be isolated. This action resulted in the closure of the regulating valve and the motor-operated isolation valve in each feedwater line. This action termi-- nated the flow of auxiliary feedwater backwards to the ruptured flash evaporator 3-7
and redirected it to the steam generators. Thus, the heat sink provided by the steam generators / emergency feedwater was degraded only temporarily. Until the motor operated valve in the feedwater piping to each steam generator was closed, the water in these pipes was draining backwards through the conden-sate system, to the leak. Draining, or emptying these pipes is supposed to be prevented by the five check valves in the feedwater system. Their specific purpose, by design and safety function, is to keep the pipes filled with water and to prevent backleakage to the condensate system, especially the auxiliary feedwater flow which is introduced into the feedwater piping downstream of the - check valves. Thus, without such isolation, the auxiliary feedwater was flowing back through the condensate system to the ruptured flash evaporator. -
- 3. 5 Conflicting Requirements A loud " boom" awoke the Shift Technical Advisor (STA) just after he had gone to sleep at about 4:30 a.m. His sleeping quarters are located in a mobile home between Units 1 and 2/3, within the site's protected area. Earlier that night he had been assisting in troubleshooting the electrical ground, primarily by interpreting and evaluating the Technical Specification requirements for the ac power supply. After the noise awoke him, he tried to contact the control room, first on his two-way radio and then using the plant telephone system. Unable to contact anyone, he got dressed and went to the control room, arriving there about 5:02 a.m. according to the control room clock. On his way he noticed steam coming from the flash evaporator. At that point, he did not know that ac power had been lost.
He assumed his normal duties after a reactor trip by exe-cuting the EDI containing the Critical Safety Function Status Trees. These are part of the emergency operating guidelines developed by the Westinghouse Owners Group and are used in monitoring those safety functions related to the mainte-nance of the various barriers that prevent the release of radioactive material to the environment. The Supervisor of Coordination, who had entered the control room during the period power was lost, asked the Shift Superintendent if it would be okay if he reset the radiation monitors which were still alarming. (Note-on a loss of power, the radiation monitors fail in a mode which isolates the containment building, including the steam generator blowdown.) Receiving permission, he reset the radiation alarms. As a result, without operator awareness, steam generator blowdown was automatically re-established at about 100 gpm per steam generator. Within a few minutes, as he monitored the status trees, the STA in-formed the Shift Superintendent that the steam generator levels were low, but not low enough to be in the alert range. - Meanwhile, in response to decreasing pressurizer water level, a reactor opera-tor started a charging pump. The second charging pump then started automatic- . ally in response to the low charging header pressure. The suction of both pumps then automatically switched to the refueling water storage tank when a low level setpoint was reached in the volume control tank. The suction to the pumps switched several times between the volume control tank and the refueling water storage tank until it was manually switched by the operators to the refueling water storage tank late in the event to borate for cold shutdown. The water level in the pressurizer was decreasing toward off-scale low, and the pressurizer pressure was decreasing toward the low pressure setpoint (1735 psig) for safety injection actuation. 3-8
Beginning at about 5:00 a.m., the operators had two conflicting interests or concerns to deal with prior to getting control of the reactor coolant pressure. First, on the primary coolant side, both the pressurizer level and reactor cool-ant temperature were low and decreasing. Second, on the secondary system, all three steam generator water levels were low and decreasing. The primary param-eters indicated an abnormally high cooldown during natural circulation conditions in the reactor coolant system--something like a steam line break--but without a steam break. The operators were concerned and discussed the situation with the Supervisor of Coordination. The operators did not want the steam generators to go dry, but at the same time, the operators wanted to minimize the heat trans-fer in the steam generators in order to recover primary system level and
. pressure.
The operators recognized that the reactor coolant system was being overcooled.
, In order to prevent the system from reaching the setpoint for safety injection actuation, a reactor operator directed another operator controlling the auxiliary feedwater system to stop the flow. Accordingly, the throttle valves were closed terminating auxiliary feedwater flow. The steam generator water levels then decreased at a faster rate. Noting this condition, the Shift Superintendent, who was maintaining an overview of control room activities, directed the oper-ator to restore auxiliary feedwater flow. Based on the recollection of the operator, auxiliary feedwater was restored to an indicated level of about 25 gpm to each steam generator in about 10 seconds.
Also during this period, a nuclear plant equipment operator was dispatched to the turbine building to manually close the two 24-inch steamline block valves to minimize the cooldown. Closing these valves is normally done after a reactor trip because of secondary-side steam leaks. Unlike other plants with remotely operated main steamline isolation valves, these valves cannot be remotely oper-ated from the control room. They must be closed by equipment operators using a large hand-held, air-operated wrench. 3.6 The Water Hammer At about 5:07 a.m., the equipment operator had just started to close one of the steamline block valves located on the turbine building mezzanine level when he heard a bang, felt a concussion wave, and was engulfed by a cloud of steam. He ran from the area, but not before the steam had soaked his clothing. When he reached the control room he reported that the main steam line had broken. The control room operators had also heard the bang. However, what the control room operators had actually heard, and what the equipment operator had witnessed, was a failed feedwater check valve caused by a thermal-hydraulic phenomenon known as water hammer.
. As the auxiliary feedwater pumps refilled the feedwater piping to the steam generators, conditions were being established for a phenomenon that can generate destructive forces greater than 150,000 pounds-force. Since the feedwater pip-ing to the steam generators had drained because of the failed check valves, the pipes contained water and steam at high temperature and pressure from the steam generators. As the auxiliary feedwater system filled the piping with relatively cold water, an instability occurred at the steam / water interface, which created a slug of water in the steam space. The slug accelerated at great speed, as steam was condensed in front of the slug, until it encountered an obstruction 3-9
or a change of direction in the piping, such as at an elbow or closed valve. Upon contact, the slug imparted its energy to the piping with the force of a hammer blow, i.e., a condensation-induced water hammer. Because of the long (203 feet) horizontal layout of the feedwater piping to the B steam generator and other sustaining conditions, this piping experienced the water hammer. The forces from the water hammer displaced the 10-inch diameter feedwater piping, distorted its original configuration, caused an 80-inch crack, and damaged pipe hangers and snubbers. In seconds, the one-half inch thick piping was irrevers-ibly damaged--the 80-inch crack, 30 percent through the wall at places, indicates how close the pipe had come to splitting open. Outside the containment building, the forces associated with the water haramer - were forceful enough to stretch 10 one-half-inch diameter bolts holding the bonnet on a 4-inch bypass check valve by about one-half inch. All of the bolts were stretched into an hour glass shape. The steam and water from the check - valve body to bonnet interface had sufficient force to blow away the insulation from all the piping located 360 degrees around the check valve. The steam es-caping through the gap between the bonnet and valve body was felt 25 feet away by the nuclear plant equipment operator who was closing the steamline block valve. The design of the steam system at Unit I has the three steamlines joined into a common pipe (or steam header) inside the containment building without any valves to prevent simultaneous blowdown of all three steam generators should a leak in a steamline or a feedwater line occur. Hence, the leak from the B feedwater bypass check valve located outside the containment building communicated with all three steam generators, via the steam header and B feedring, and their steam inventories were vented via the leak to the atmosphere. In addition, the auxil-iary feedwater flow to B steam generator escaped from this leak instead of going to the steam generator. 3.7 Steam Generators Boil Dry The effects of the water hammer and the failed bypass check valve were not indi-cated in the control room. Based on the report by the nuclear plant equipment operator, the control room operators thought that there was a steamline break. However, they continued to follow the E0I for reactor trip response in a system-atic manner because their instrumentation did not reflect a major steamline break. With both charging pumps operating at full flow, the reactor coolant pressure and pressurizer level recovered, and control of primary pressure was regained. At this point, the E0I required that the B reactor coolant pump be started for pressurizer spray control. Shortly after starting the pump, a thrust bearing high temperature alarm sounded. Although the operators believed it was a false . indication. discussions with the Supervisor of Coordination led to a decision 20 minutes later to start the A and C pumps and shutdown the B pump. When the two reactor coolant pumps were started, the steam generator levels were about equal, but low on the wide range level indicators. The operators recalled that the level went off-scale low in all three steam generators shortly after the A and C pumps were started. The Shift Superintendent noticed a sharp decrease in steam pressure, and recalling earlier the report by the plant equip-ment operator, declared he thought that there was a steam leak. The auxiliary 3-10
i feedwater flow rate was maintained at about 25 gpm (indicated) to each steam generator in order to assure the heat sink was maintained, although there was no indicated level in any steam generator. All three steam generators were essentially dry; however, the conditions for a " dry" steam generator in the E0I were not satisfied. The Control Room Supervisor reviewed the E0I for loss of secondary coolant but, based on secondary system conditions, the criteria were not met for beginning the procedure (e.g. , steam pressure was above 400 psig). The E0I for responding to a steam generator low level has three conditions that must exist simultaneously for a steam generator to be considered dry. If either the wide range water level indicates greater than zero, or the reactor coolant loop temperature difference is greater than zero, or if auxiliary feedwater flow to the steam generator is 25 gpm or more, the steam generator is considered to be effective in removing decay heat and is not considered dry. The operators ' did not recall any time during the event that at least one of the three condi-tions did not exist. However, none of these parameters are recorded on a strip chart. The STA continued to cycle through the E01 for the Critical Safety Function Status Trees. He also maintained a vigilant watch on the steam generator levels. Because the narrow range level indication was below 10 percent, the E0I referred him to the E0I for responding to steam generator low level. As he went through the E01, step 3 required that the steam generator blowdown be isolated-- a significant discovery. Prior to this time, about 100 gpm had been draining from each steam generator as blowdown. This was the dominant contributor to the loss of water inventory and level in the steam generators. The STA informed the Control Room Supervisor of this important discovery, and the blowdown was secured at about 35 minutes after it had automatically been re established when the radiation monitor alarm had been reset. The continuous blowdown from the steam generators had escaped recognition by the operators in analyzing the reasons for the coo?down and steam generator low levels. (The status of the blowdown system is not indicated in the control room.) By this time, it was obvious to the Shift Superintendent that the plant had to be placed in cold shutdown in order to isolate and correct the leak. At this point, the control room operators did not know that a water hammer had occurred and that it had caused the steam leak outside containment and severe damage to feedwater piping and hangers inside the containment building. . 3.8 Plant Cooldown Another discussion took place in the control room between the operators and the , Supervisor of Coordination concerning how best to cool the reactor to cold shut-down. Although the procedures called for a normal 50 percent level in the steam generators, the operators concluded that restoring such a high level would ex-acerbate the cooldown transient, and could result in exceeding the Technical Specification cooldown rate of 100-degrees Fahrenheit per hour. All steps in the E0I for a reactor trip response had been completed and the final step re-ferred the operators to the standard operating procedure for changing operating modes from power to hot standby. Reactor coolant conditions indicated that they had passed through this procedure already, and that it would be appropriate to go to cold shutdown from hot standby. 3-11
The operators decided to maximize the cooldown rate in order to depressurize quickly and isolate the leak while maintaining sufficient margin to the Techni-cal Specification limit. Consequently, the auxiliary feedwater flow rate was increased to the A and C steam generators from about 25 to 40 gpm indicated. The flow rate to B was not increased because the B reactor coolant pump had been secured, minimizing the primary-to secondary heat transfer and the need for additional feedwater. The levels in the A and C steam generators increased slowly, but level in the B steam generator remained off-scale low. It was then apparent to the operators that the leak was associated with the B steam gener-ator. At this point, they established a cooldown rate of about 60-degrees Fahrenheit per hour and started the preparations that had to be completed prior to establishing decay heat removal using the residual heat removal (RHR) system. - Based on the STA's review of the Critical Function Status Trees, he noticed that containment building pressure was slightly positive when it should have been - negative. He so informed the Shift Superintendent, who then determined that the line which normally vents air from the containment building had berti isolated by the radiation monitor when power was lost. Thus, the leakage from air-operated valves inside the building had caused the pressure increase. Earlier during the event, the saltwater cooling system had been aligned to the heat exchangers in the turbine plant cooling system when containment building cooling was re-established. The RHR system requires the full capacity of the saltwater cooling system for a rapid cooldown. Thus, the saltwater flow to the turbine plant cooling system had to be terminated and redirected to the RHR system. An alternate arrangelr.ent involving the intake screen wash pumps was then used to supply cooling water to the turbine plant cooling water heat exchangers. The normal cooling water to the turbine plant cooling water heat exchangers is provided by the circulating water system. However, numerous attempts to estab-lish the conditions for starting the circulating water pumps failed due to abnormal conditions in the main condenser. The temperature in the condenser was about 200 degrees Fahrenheit (normal is about 100 F). When a vacuum was applied to the tube side of the condenser to ensure that the water box was filled, the circulating water would flash to steam, and the vacuum interlock for starting the circulating water pumps could not be satisfied. Evidently, the steam traps and other secondary sources of hot water ard steam overheated the condenser in the absence of circulating cooling water. At the time, the operators did not understand why the tubes could not be filled with water. One of the last preparations made prior to opening the isolation valves to the - RHR system was a containment building entry to close the valve in the bypass line around a RHR pump. The bypass line provides an alternate hot leg injection path following a loss of coolant accident. Closing the valve is a normal oper- . ation and ensures the maximum flow rate through the RHR heat exchangers, i.e. , faster cooldown. At about 9:20 a.m., all preparations had been made and the operators unsuccess-fully attempted to open the suction and discharge valves that isolate the RHR system from the reactor coolant system. The valves are interlocked such that the reactor coolant pressure must be well below the design pressure of the RHR system before the valves will open. The operators investigated and reviewed 3-12
l l 1 the as-built drawings and confirmed that the pressure setpoint was 400 psig. The reactor coolant system pressure was about 370 psig at the time. The oper-ators assumed the interlock had failed and overrode the relay in the interlock logic and opened the valves. Later it was found that a procedural deficiency and related training misled the operators to believe that the interlock had failed. At 9:41 a.m., about 5 hours after the event started, RHR cooling was established and the Emergency Coordinator terminated the Unusual Event. The major task re-maining was isolating the leak from the failed check valve. 3.9 Isolating the Leak At about 6:00 a.m. , the day shift operating crew began arriving at the control room. Station management, engineers, NRC Resident Inspectors, and other person-nel also came to the control roon The shift turnover that normally begins at 7:00 a.m. (and is usually completed by 7:30 a.m.) did not occur until 10:00 a.m. because of the event. The day shift and most of the other personnel re-mained outside the control room in the Technical Support Center and supported the recovery operations, including the isolation of the leak. Attempts to identify the location of the leak began about 5:30 a.m. An Assist-ant Control Operator dressed in a steam suit (i.e., protective clothing for environmental protection against fires, steam, hazardous materials, etc.) in-spected the mezzanine area. Two members of the fire brigade accompanied him into the area, which was standard procedure for rescue operations in a hostile environment. Radiological surveys had previously been completed by two health physicists. The first entry in the steam environment lasted only about 2 minutes because of the heat. The second entry was made with additional protective clothing. The operator was then able to determine that the leak was near the feedwater bypass regulating valve. But his visibility and mobility were so restricted that he did not attempt to isolate the leak. He returned to the control room at about 8:00 a.m., and informed the Shift Superintendent of his findings. Based on the status of the plant, the operators concluded that there was not an urgent need to isolate the leak. The conditions in the reactor coolant system were stable, and water levels had been reestablished in steam generators A and C. The leak was effectively removing decay heat, with a resultant steady cooldown rate. At 10:45 a.m., the manual valve in the B steam generator main feedwater line and a manual valve in the bypass piping were closed, which isolated the leak. , The auxiliary feedwater system refilled B steam generator. The unit entered mode 5 for a refueling outage at about 3:00 p.m.--a week sooner than planned. After the shift change, operators that had been on duty were debriefed by site management to ascertain the sequence of events. During the debriefing, it be-came apparent to the operators and management that a water hammer had occurred, and that an inspection of the systems inside the containment building was required. It was during this containment entry, early the next day that operators found evidence of damage to the feedwater piping to the B steam generator. The 3-13
inspection revealed displaced pipe, missing and damaged insulation, and damaged pipe supports resulting from thermal hydraulic forces not previously considered in the design of that piping. Thus, what had begun early in the shift as an attempt to isolate an electrical problem, led at 4:51 a.m. to a temporary loss of inplant ac power, a condition which, combined with five failed check valves, subsequently resulted in an incidence of water hammer powerful enough to challenge the integrity of the safety-related feedwater system. 9 9 O O 3-14
Table 3.1 Chronological Sequence of Events Initial Conditions at Unit 1, November 21, 1985
- Saltwater leaking into the main condenser at 5 x 10 -3 g, l - Unit operating at 60 percent reactor power to facilitate search for condenser leak - South circulating water pump shut down to allow entry into south condenser water boxes
. - Steam generator blowdown ongoing at about 100 gpm per generator to minimize chloride buildup
- Electrical ground troubleshooting in progress; ground determined to be located on auxiliary transformer C supply to 4kV bus IC - Bus IC power supply shifted to 4kV bus 1A, powered from the output of the main generator through auxiliary transformer A - Auxiliary transformer C remained energized, supplying power to 4kV bus 2C, while personnel inspected electrical equipment Fox 3 critical function monitor system recording function disabled because of previous power interruption during ground isolation effort Transient Initiator 04:51:11 Auxiliary transformer C differential relays detected a phase-to phase fault current in excess of 1500 amps and actuated trips in associated circuit breakers to isolate the transformer.
Circuit breakers 4032 and 6032 opened to isolate auxiliary trans-former C from the 220kV switchyard. Circuit breaker 12C02 opened to isolate auxiliary transformer C from 4kV bus 2C. - Systems Response / Operator Actions 04:51:11+ Bus 2C de-energized, de-energizing the following selected loads: O East feedwater pump Southeast condensate pump Northeast condensate pump l East heater drain pt=p Vital 120VAC bus 4 ENS phone began to ring along with all the other alarms associated j with the trip of the auxiliary transformer as the Shift Supervisor ; entered the control room. Il 3 3-15
Table 3.1 Chronological Sequence of Events Diesel generator 2 started automatically on loss of 4kV bus 2C, but did not load automatically, per design. East feedwater pump discharge check valve failed to seat as the de energized pump coasted down. Running west feedwater pump pressurized the east condensate-feedwater heater train. East flash evaporator condenser tubes became overpressured, ruptured and overpressurized the evaporator shell, causing the shell to develop a fishmouth opening approximately 20 feet long and 2 feet - wide. The accompanying noise was described as a " muffled howitzer." 04:51:31 Operators manually tripped the reactor in response to loss of vital 120VAC bus 4, as required by procedure, due to wholesale loss of control room instrumentation. The reactor trip initiated a turbine trip. 04:51:32 Operators pushed the unit trip button, opening main transformer out-put circuit breakers 4012 and 6012, auxiliary transformer A and B output circuit breakers 11A04 and 11804, and tripping the turbine. 04:51:32+ All inplant power was lost, except for 120VAC vital buses carried by inverters. All inplant lighting was lost, except for battery powered emergency lighting. Letdown, steam generator blowdown and the containment sphere mini-purge isolation valves shut on loss of power. Diesel generator 1 started automatically on loss of 4kV bus IC, but did not load automatically, per design. Station loss-of-voltage automatic transfer scheme initiated to allow backfeed of offsite power through the main and auxiliary transformers. Security access control equipment malfunctioned following automatic transfer to alternate power supply. Electric and steam powered auxiliary feedwater pumps received auto-matic initiation signals on low steam generator level, due to level drop following reactor trip and turbine stop valve closure. The electric-driven pump started later, after electric power was re-stored. The steam turbine-driven pump began a 3 -minute warmup period. All three steam generator feed regulating valves shut to 5 percent flow position in automatic response to a reactor trip. 3-16
I I Table 3.1 Chronological Sequence of Events As the west feedwater pump stopped, its discharge check valve and l the check valve downstream of the regulating valve of the C steam l generator failed to seat. At the same time, the discs in each check valve downstream of regulating valves to A and B steam generators settled to the bottom of their respective valve bodies. All three steam generators began to empty their feedwater lines to the east flash evaporator condenser because of the tube rupture. Shift Superintendent picked up spuriously ringing ENS phone, informed the NRC Headquarters Duty Officer (HQDO) of the reactor trip and loss of power, promised to call back, responded to questions, stated that , offsite power was available and that the plant was stable and tripped, and again promised to call right back. Operators verified that rod bottom lights energized, indicating the reactor had tripped. East and west main feedwater pump shaft seal drain trap vents were observed to be blowing excessive steam and water. The fire watch in the 4kV switchgear room received a fire alarm from the lube oil reservoir area, observed steam in the area and called station emergency services. East condensate-feedwater train condensate relief was observed to be blowing steam. Main feedwater pump suction and discharge temperatures increased to approximately 400 F. Operators responded to a spurious annunciation and sequencer light indication of initiation of the safety injection system, but determined that plant parameters did not require operation of the system and that the system had, in fact, not actuated. Station emergency services dispatched a fire truck to Unit 1. Operators observed that the 18kV system isolation light actuated, . indicating that the first phase of loss of voltage auto transfer scheme had been completed. , Operators attempted to reset the unit trip lockup bus to enable back-feed of power from the switchyard, but the reset failed, apparently due to the timing of the attempt before the main generator no-load motor-operated disconnect was fully opened. The operator did not verify that the reset was effective. Operators found security access controls were not responsive and utilized planned procedures, personnel, and hardware to compensate. 04:55+ Steam turbine-driven auxiliary feedwater pump completed its warmup cycle and began to deliver approximately 130 gpm AFW flow (indicated 3-17
Table 3.1 Chronological Sequence of Events flow was about 110 gpm/SG) at outside ambient temperature to main feedwater lines just downstream of the thrce feedwater control sta-tions. Reverse flow in the main feedwater line carried AFW to the condensate system. Operators decided that the station loss of voltage automatic transfer scheme had failed and attempted to complete the sequence from the control room. Operators discussed energizing buses using the running but unloaded diesel generators. Operators decided to energize buses using the preferred offsite power source. . The first attempt to close 220kV switchyard circuit breaker 4012 failed because an operator did not push the synchronizing check-bypass pushbutton. 04:55:13 The second attempt to close 4012 succeeded when the operator correctly depressed the pushbutton, but it immediately tripped free because the unit trip lockup bus had not been reset. 04:55:15 The third attempt to close 4012 had the same results as the second attempt. An operator reset the unit trip lockup bus. The first attempt to close 220kV switchyard circuit breaker 6012 failed because an operator had again not depressed the synchronizing check-bypass pushbutton. 04:55:24 The second attempt to close 6012 succeeded, backfeeding power from the 220kV switchyard, which had remained energized, to auxiliary transformers A and B. Operators closed the feeder circuit breaker from auxiliary trans-former A to 4kV bus IA, re-energizing 4kV bus 1A and IC. (The tie breaker between bus 1A and IC had never been opened.) Operators closed the feeder circuit breaker from auxiliary trans- . former 8 to 4kV bus 18 and from bus 18 to 2C. Operators subsequently completed re-energization of the station by powering the remaining de-energized 480VAC buses. , The electric powered auxiliary feedwater pump started with a 20-second delay upon regaining power, due to the continued presence of a steam generator low level signal, and increased AFW flow to approximately 155 gpm per steam generator (indicated flow was about 135 gpm/SG). Letdown automatically reinitiated on return of power, but the charging pumps remained tripped. 3-18
l Table 3.1 Chronological Sequence of Events Atmospheric steam dumps actuated on return of power, but operators shifted steam dump operations to automatic pressure control, thereby { securing steam dumps. Operators shut feedwater isolation valves MOV-20, 21 and 22 and feedwater regulating valves FCV 456, 457 and 458, as required by procedure, unknowingly stopping further voiding of steam generator feedwater lines and starting the refilling process at a rate of , approximately 155 gpm per steam generator. The Supervisor of Coordination reset radiation monitor alarms that
- were received because of loss of power. Resetting the monitor for steam generator blowdown re-initiated blowdown for each steam genera-tor at about 100 gpm.
Letdown isolated automatically on low pressurizer level. Operators checked pressurizer level and pressure as required by pro-cedure, found level and pressure were low and decreasing, at about 5 percent and 1880 psig, respectively, and became concerned that plant cooldown could be excessive or cause safety injection. 04:58 Operators started the south charging pump to raise pressurizer level. 04:59 The north charging pump started automatically on low charging header pressure with one charging pump running. 05:00 The suction of both charging pumps shifted automatically between VCT and RWST and back as the level cycled through VCT low level set points. Operators verify proper operation of AFW pumps. 05:01 The Shift Supervisor called HQD0 on ENS to provide information on the plant transient. He completed a call 2 minutes later, indicating that they would probably declare an alert and close it out in the same call, that he needed to go, that he was still dealing with the problem and would call back. . Operators started reactor coolant pump B to provide a source for pressurizer sprays for pressure control. 05:02 Operators terminated AFW flow to the steam generators to minimize RCS cooldown; then subsequently resurred AFW flow to all steam generators at a rate of about 40 gpm per generator (indicated flow was about 25 gpm/SG). The STA arrived in the control room. l A plant equipment operator was dispatched to manually close main steam block valves to reduce plant cooldown. 3-19
Table 3.1 Chronological Sequence of Events 05:06 An unusual event was declared onsite. (Licensee Emergency Plan Tab D-1-1.) State and local offsite agencies were informed. A Prompt Notification Report of the declaration of an Unusual Event was not made to NRC. 05:07 A loud " bang" was heard. The nuclear plant equipment operator, sent to shut the main steam block valves, heard a water hammer and observed - steam on the turbine building mezzanine. The operator left the mezza-nine without shutting the main steam valves. - 05:08 Circuit breaker 4012 was closed by an operator utilizing the synchronizing check-bypass pushbutton. - 05:09 The reactor cooling pump B thrust bearing high temperature alarm annunciated. 05:10 The control room received a report of a steam leak on the feedwater mezzanine from a dripping wet operator, who had just returned from that location. Letdown valves opened after pressurizer level rose above 10 percent. 05:12 Operators shut the turbine plant cooling water (TBCW) supply valve for containment sphere air coolers and started a TBCW pump. An operator was dispatched to re-establish TBCW flow to containment sphere air coolers. 05:17 Charging pump suction was shifted to the RWST to start boration for cold shutdown. 05:20 Operators reset the safeguards sequencers and secured the unloaded diesel generators. Operators secured the lube oil reservoir foam system and fire pump, after confirming that the system should not have actuated. 05:24 Operators started reactor coolant pump A. 05:27 Operators started reactor coolant pump C. - The wide range level indication dropped off-scale low in all three steam generators. , j 05:28 Operators stopped reactor coolant pump B. l l Operators decided to establish rapid controlled cooldown of RCS at I about 100 F/hr to stop the assumed steam leak. l Operators increased flow to steam generator A and C from about 40 gpm to about 70 gpm each. AFW flow to the B steam generator was maintained at about 40 gpm. 1 3-20
i Table 3.1 Chr.inological Sequence of Events 05:30 Blowdown from steam generators was secured by reducing the setpoint on the radiatiori monitor. l Wide range wster level indication returned on-scale in A and C steam generators. Operators commenced periodic air sampling for radioactivity in the vicinity of the steam leak. The highest sample reported showed , 5 X 10 10 uCi/cc. Personnel wearing steam suits made two attempts to identify and , secure the source of the steam leak. 05:45 The turbine generator was placed on a turning gear. Operators shut steam generator blowdown micro-valves. HQD0 called SONGS-1 on the ENS to check plant status and establish an open line. The shift superintendent was asked to call back once he could get someone assigned to maintain an open line. 05:46 Safety injection was blocked. The north charging pump was secured. unknown Sandbags were placed at the entrance to the chemical feed room to prevent water from flowing across the floor into the electrical switchgear rooms. 05:48 SCE's emergency coordinator called HQDO, was persuaded of the need to establish an open ENS line to NRC, but was told of phone problems and that he would be called back. 05:57 Operators stopped boration using RWST. 05:58 Operators noted containment sphere pressure was slightly positive; found that containment sphere mini purge valve CV-10 had not been re-opened after the radiation monitors were reset following resto"- ation of power; and opened CV-10, allowing containment sphere prea- . sure to return to its normal, slightly negative condition. HQD0 succeeded in establishing an open line between the site , emergency coordinator, the NRC regional duty officer and the headquarters Incident Response Center. The line would remain open until released by NRC. The HQD0 notified FEMA of the declaration of an Alert. 06:15 Operators, unable to start the circulating water pumps due to high condenser temperature and steam in the water boxes, aligned saltwater cooling to the turbine plant cooling water heat exchangers. 06:30 Operators started emergency boration for cold shutdown. 3 .-
Table 3.1 Chronological Sequence of Events 07:00 Operators secured emergency boration. 07:02 The Plant Superintendent assumed the role of emergency coordinator and changed the Unusual Event declaration basis to Emergency Plan Tab. G-1-2. 07:44 A monitoring team dispatched to measure potential offsite radio-activity determined downwind site boundary radiation levels to be less than 0.1 mrem /hr. - 07:47 Operators started the second emergency boration to assure 5 percent shutdown in mode 5. - 07:55 Emergency boration was secured. 08:00 Entered Mode 4; operators still believed there was a steam leak. 08:35 Operators secured the turbine-driven auxiliary feedwater pump due to low steam pressure. 08:36 Operators aligned screen wash pumps to supply cooling for the tarbine plant cooling water heat exchangers. 08:37 Operators aligned salt water cooling pumps to provide maximum component cooling water heat exchanger cooling. 09:00 Operators started a third component cooling water pump iri preparation for initiating RHR. 09:10 Operators a tempted to open RHR suction valves, M0r'Q]i and 834, but pressure interlock had not yet cleared, although RCS pressure was well below 400 psig. Air sample from the chemistry sample room determined that noble gas activity was 1.87 times the maximum permis6ible level. 09:12 Containment sphere entry was made to isolate the hot leg recircula-tion flow path by shutting RHR-004. 09:18 Operators overrode the high pressure interlock and opened MOV-813 and 834. 09:20 Operators stopped vacuum pumps. 09:30 Operators shut RHR-004. 09:35 Operators started the West RHR pump. , 09:38 Operators started the East RHR pump. 09:40 The Unusual Event was terminated and both RHR pumps were in service. l 3-22
Table 3.1 Chronological Sequence of Events 10:00 Shift turnover began and continued sequentially until all positions were briefed and properly relieved. 10:45 Feedwater leakage was manually isolated. 11:15 A work order was issued to repair the security system affected by moisture from the leak. 13:20 SteamgeneratorsamplesshowedactivityinAandClessghanthe threshold of detectability; activity in B was 2.87 x 10 uCi/ml. 14:06 Operators restarted the 480V room air conditioner. 14:10 Operators isolated a dc ground on control power to FCV-456 and 457. 14:36 Operators commenced RCS degassing. 15:08 The plant entered Mode 5. NOVEMBER 22, 1985 01:00 Operators entered the containment sphere and identified damaged pipe supports and insulation on the B steam generator feedwater line. 16:41 Operators secured electric auxiliary feedwater pump. 17:32 Operators secured filling the AFW tank from Unit 2 and 3. 21:40 Operators manually closed the main steam isolation valves. 22:45 Operators transferred water from A to B steam generator, using the blowdown lines. O e 3-23
4 DESCRIPTION OF PLANT SYSTEMS 4.1 General Design The nuclear steam supply system (NSSS) for the San Onofre Nuclear Generating Station, Unit 1, is a pressurized water reactor (PWR) supplied by the Westinghouse Electric Company. The plant was designed and constructed by the Bechtel Power Corporation. Licensed power level is 1337 MW(t), yielding a net electrical output of approximately 450 MW(e). The reactor coolant system (RCS) consists of a reactor pressure vessel and
. three parallel heat transfer loops. Each loop contains a model 27, U-tube steam generator and a reactor coolant pump. RCS pressure control and overpressure protection are provided by a pressurizer connected to loop B. , Pressurizer spray is provided from loops A and B. Normal operating pressure is 2085 psig.
As corrective action for steam generator tube corrosion, approximately 8.6 percent of the tubes have been plugged and approximately 57 percent have been sleeved. This plugging and sleeving has resulted in lower-than-design RCS flow and reduced steam generator effective heat transfer surface. Further, to pro-vide a less hostile operat ing environment for the tubes, RCS temperature was reduced from design temperature. Control and protection setpoints were adjusted to accommodate these changes in operating temperature and flow. Current pro-grammed average RCS temperature at 100 percent power is 551.5 F. A self-imposed limit by SCE restricts power to 93 percent based on steam generator moisture carryover considerations. 4.2 Main Steam System The main steam system distributes steam from the three steam generators to the high pressure turbine, moisture separator-reheaters, turbine-driven auxiliary feedwater pump, steam dump and other miscellaneous loads. As shown in Figure 4.1, the steam generators discharge into a common ring header within the containment building, which then connects to the two main steam lines. The main steam lines penetrate the containment building and distribute steam to the main turbine and other loads. Each main steam line has a manually operated (air wrench) block valve (MSS 301 and 302), which is used for maintenance isolation and to limit cooldown rates by isolating leaky valves and steam traps after plant shutdown. A branch line is connected to each main steam line, outside of the containment
- building, upstream of the manual block valves. Each branch line contains five I
. self-actuating code safety valves and two steam dumps which discharge to the atmosphere. The pneumatically operated atmospheric steam dumps are controlled by the steam dump system in response to: (1) reactor coolant temperature during load rejections or turbine trips and (2) steam pressure during startup or shut-down operations. Steam to the turbine-driven auxiliary feedwater pump comes from one of the main steam lines outside of the containment building upstream of the manual block valve.
4-1 1
Connections to the main steam lines between the manual block valves and turbine stop valves include a cross-connect line and supply lines to the four moisture sep6rator-reheaters and condenser steam dump valves CV 3 and 4. The condenser steam dump valves are operated by the steam dump system in response to the same control signals as the atmospheric steam dumps, except that they are automati-cally disabled on low condenser vacuum. Since there are no check valves or automatic isolation valves to isolate the steam generators from each other, a steam or feedwater leak can affect all three steam generators. ~ 4.3 Steam Generator Blowdown System - The steam generator blowdown system continucusly removes (blows down) water from the steam generators. This blowdown helps maintain the purity of steam - generator water by removing contaminants (both chemical and radiochemical) which tend to concentrate in the steam generators. These contaminants could contribute to steam generator tube degradation. The blowdown flowpath shown in Figure 4-2 runs from the steam generator bottom blowdown connections, out of the containment bui'lding, through manual' angle throttling valves FWS 512, 516, and 521, into a common header. Flow from the common header runs through remotely operated angle check valve CV 100 to the blowdown tank. Flow from the blowdown tank can be directed either to the cir-culating water discharge or to the liquid radwaste holdup tanks, depending upon the activity level of the blowdown. An alternate flowpath bypasses the blow-down tank through CV 100 A and B directly to the circulating water outfall. Radiation monitor R-1216 continuously samples blowdown water. The monitor normally analyzes a common sample from all three steam generators but may be realigned to sample each steam generator individually. If R-1216 senses high radiation or loses control power, it alarms in the control room and automati-cally isolates blowdown by closing CV 100, 100A and 1008. 4.4 Main Feedwater System The main feedwater system normally provides heated feedwater at a controlled rate to the three steam generators. The auxiliary feedwater system (Section 4.6) connects to the main feedwater piping to each steam generator just prior to their containment building penetrations. In its normal mode, main feedwater is not a safety system. Because the two main feedwater pumps are part of the safety injection flowpath (Section 4.11), the pumps and all equipment required - to realign their suctions and dischsrges are safety-related. As shown in Figure 4.3, each main feedwater pump receives flow from a separate . train of low pressure feedwater heaters in the condensate system through a pneumatic / hydraulic isolation valve. Each feedwater pump discharges through a check valve and a pneumatic / hydraulic isolation valve to a first point (high pressure) feedwater heater. The first point heater outlets join into a common header to supply the feedwater control stations for the three steam generators. Flow to each steam generator is through an individual control station consisting of a motor-operated isolation valve, an air-operated main feed regulating valve, a check valve and a manually operated isolation valve. A small bypass line containing manual isolation valves, an air-operated bypass flow control valve, 4-2
and a check valve is used for low flow conditions such as startup and shutdown operations. Each main feedwater line from the motor-operated isolation valves to the steam generator is seismically qualified. The main feedwater pumps are powered by 3,500 hp electric motors supplied from 4kV buses IC and 2C. Seal water is supplied from the discharge of the conden-sate pumps. Main feedwater pump discharge check valves FWS 438 and 439 prevent reverse flow through an idle pump. These and the other main feedwater system check valves are manufactured by MCC Pacific. The first point or high pressure heaters receive extraction steam through
, nonreturn check valves from the high pressure turbine. Tube side design pressure is 1367 psig. Each first point heater has a shell side relief which discharges to the blowdown tank.
The following feedwater system parameters are indicated and recorded in the control room:
- 1. Main feedwater pump suction pressures and temperatures.
- 2. Pressure and temperature in the common portion of the main feedwater system at the outlet of the first point heaters.
- 3. Feed flow, steam flow and level for each steam generator.
After a reactor trip, main feedwater regulating valves FCV 456, 457 and 458 are automatically throttled to five percent flow, which is sufficient for decay heat removal, but minimizes overcooling due to excessive feeowater addition. The reactor trip response procedure directs the operators to isolate the main feedwater flowpath, after verifying auxiliary feedwater flow, by closing the main and bypass regulating valves (FCV 456, 457, 458, 142, 143 and 144) and the motor-operated feedwater header isolation valves (MOV 20, 21 and 22). A safety injection system signal automatically shuts the main feedwater regulating valves, the bypass valves and the motor-operated isolation valves. 4.5 Condensate System The condensate system transfers deaerated condensate from the main condenser hotwells through two parallel trains of low pressure feedwater heaters to the suction of the main feedwater pumps. Steam extracted from the main turbine provides the heat source for the feedwater heaters.
. As shown in Figure 4.4, the system begins with the four condensate pumps which take suction from the hotwells of condensers E-2A (north) and E-28 (south).
The four hotwells are tied together by a 24-inch line which also connects to
, the suctions of all condensate pumps. The discharges of the pumps are tied together after their respective check valves and then the system splits into two trains. Each train consists of:
- 1. An air ejector condenser.
- 2. A turbine gland sealing steam condenser.
- 3. A flash evaporator unit containing a flash evaporator, a 5th point heater and drain cooler and a 4th point heater and drain cooler.
- 4. A 3rd point heater.
- 5. A 2nd point heater.
4-3 l
The system is nonsafety-related up to the pneumatic-hydraulic isolation valves in the main feedwater pump suctions. The condensate pumps are tripped on loss of voltage on bus 1C and 2C or a l safety irijection system signal. Pump seal water is provided from the discharge ! of the condensate pumps. The air ejector and gland steam condensers are shell and tube heat exchangers with condensate flowing through the tube side. Tube side design pressure is 350 psig. The flash evaporators are combined units consisting of a flash evaporator in a - common housing with the 5th and 4th point low pressure feedwater heaters and drain coolers. The flash evaporators are designed to produce fresh water from sea watsr but have not been used for several years. Condensate does, however, still flow through the tubes of the evaporator condenser. Design pressure of the tube side of the flash evaporator condenser and low pressure feedwater heaters is 350 psig and shell side design pressure is 15 psig. A shell side relief valve on each flash evaporator set at a nominal 15 psig discharges to the blowdown tank. The 2nd and 3rd point feedwater heaters are also designed for 350 psig tube pressure. Shell side reliefs from the 2nd and 3rd point heaters discharge to the blowdown tank. A water box thermal expansion relief valve is provided on each 3rd point feedwater heater and discharges to an open drain line and to the atmosphere under the 3rd point heater. Nonreturn check valves are provided in the extraction steam lines for all feedwater harters except the 5th point. 4.6 Auxiliary Feedwater System The auxiliary feedwater system (AFW) is a safety system designed to assure a reliabic heat sink for the reactor during abnormal or emergency conditions when mdin feedwater is not available. The AFW system also provides feedwater to the steam generators during normal startup, shutdown and hot standby conditions. Either of the redundant AFW pumps can supply sufficient flow to the steam gen-erators for decay heat removal present 3 1/2 minutes after a trip from full power. The AFW system shown on Figure 4.5 consists of a motor-driven pump, a turbine-driven pump, an auxiliary feedwater tank, and associated piping, valves and instrumentation. Flow is from the AFW storage tank through individual suction lines to the two auxiliary feedwater pumps. The pumps discharge throuch inde- . pendent check and isolation valves to the auxiliary feedwater flow control valves for the three steam generators. Each flow control valve can receive flow from the motor-driven and the turbine-driven auxiliary feedwater pump discharge header. , FCV 2300 and 3300 control AFW flow to the A and C steam generators, respectively, while FCV 2301 and 3301 control flow to the B steam generator. Each flow con-trol valve discharges through check and isolation valves to the main feedwater piping of its respective steam generator between the main feedwater control stations and the feed line containment building penetration. Auxiliary feed-water flow is then through the main feedwater piping into the steam generators. The system is automatically actuated if narrow range level in two of three steam generators indicates less than 5 percent. Narrow range level is expected 4-4
to shrink to less than 5 percent after a trip from full power. The system may also be initiated manually. The auxiliary feedwater tank is a seismically qualified 240,000 gallon tank with 150,000 gallons reserved for AFW. All non-AFW penetrations are above the 150,000 gallon level. Normal makeup is from the Unit 2 makeup demineralizer or the Unit 2 condensate storage tank and requires operator action to install the requisite fire hoses to provide a flow path. The turbine-driven auxiliary feedwater pump is a centrifugal pump driven by a single-stage turbine supplied by the turbine division of the Worthington . Corporation. Steam for the turbine is from the west main steam header, upstream of the maintenance shutoff valve, and the turbine exhausts to the atmosphere. The pump will provide a 300 gpm flow at a design head of 1093 psig. The turbine is provided with an automatic 3 1/2 minute startuo sequence to remove any water from the steam supply line and turbine casing. Flow from this pump is not available until the sequence is complete. The pump is automatically tripped on low suction pressure. Suction and discharge pressures are indicated in the control room. The motor-driven auxiliary feedwater pump is a centrifugal pump driven by a 250-hp electric motor powered from 480 volt bus 1, which can be powered by the emergency diesel generators. The capacity of the motor-driven pump is 235 gpm at 1058 psig. Pump trips include low suction-pressure, motor overcurrent or loss of power and low discharge pressure, the latter to protect the motor from pump runout. Suction pressure, discharge pressure and motor amps are indicated in the control room. Auxiliary feedwater flow control valves FCV 2300, 3300, 2301 and 3301 are divided into two trains. Train A consists of FCV 2300 and 2301 and train B consists of FCV 3200 and 3301. FCV 2300 and 3300 to steam generators A and C are manually preset to 100 percent open and FCV 2301 and 3301 to steam generator B are manually preset at 50 percent open. These settings are designed to limit AFW flow at normal steam generator operating pressures to less than 150 gpm per steam generator to minimize the potential for steam generator water hammer. Auxiliary feedwater flow to each steam generator is indicated in the concrol room. 4.7 Salt Water Cooling Systems The salt water cooling systems shown on Figure 4.6 provide a heat sink for both . safety and nonsafety-related loads. The systems consist of the condenser and circulating water system, the screen wash and salt water cooling pumps, and the turbine piant and component cooling heat exchangers. The nonsafety-related condenser and circulating water system provides a heat sink to cool and condense exhaust steam from the low pressure turbines and is the normal cooling supply to the turbine plant cooling water heat exchangers (see Section 4.8). Water from the Pacific Ocean is drawn into the intake structure, where traveling screens prevent trash and debris from entering the pumps. The circulating water pumps take suction from the intake structure and discharge through the four condenser water boxes and back to the ocean through the outfall piping. During normal operation, the north pump supplies the north waterboxes of both condenser sections and the south pump supplies the south 4-5
waterboxes. The system is designed to allow reduced power operation with one circulating water pump off. Under these conditions, either the north or south waterboxes may be isolated and drained for perscnnel entry to identify and repair saltwater leaks. Some of the circulating water pump fluw is normally diverted to cool the turbine building cooling water heat exchangers. The circu-lating water pumps have a design capacity of 173,000 gpm each and are powered by 1500-hp electric motors. The north pump is supplied from 4kV bus 1C and south pump from bus 2C. The condenser waterboxes are connected to a vacuum priming system to ensure that they are full before starting a circulating water pump. Pressure switches PS 99 and 100 prevent the start of the circulating water pumps unless they sense at least a 20-inch vacuum indicating that the waterboxes are full. - Normal supply to the nonsafety-related turbine plant cooling water heat exchangers is from the circulating water pumps through power-operated valves . (POV) 7 and 8. If the circulating water pumps are not available, the turbine plant cooling water heat exchangers can be supplied from either the safety-related saltwater cooling pumps or the nonsafety-related screen wash pumps shown in Figure 4.7. Turbine plant cooling water is important because, in addition to normal secondary plant cooling loads, it is the cooling water supply to the containment building cooling units. The safety-related component cooling water heat exchangers are normally supplied by the salt water cooling pumps. The saltwater cooling pumps are automatically started on a safety injection signal. The pumps are powered by 100-hp electric motors supplied from 480V buses 1 and 2. If the saltwater cooling pumps are unavailable, the screen wash pumps can be manually aligned as a backup. The normal function of the screen wash pumps is to backwash the traveling screens of the intake structure to wash off any debris or trash which has accumulated and to minimize differential pressure across the screens. The pumps are powered by 100-hp motors supplied from 480V buses 1 and 2. The screen wash pumps can be manually aligned to provide a backup source of cooling water to the component cooling water and the turbine plant cooling water heat exchangers. 4.8 Turbine Plant Cooling Water System The turbine plant cooling water (TPCW) system, shown on Figure 4.7, is a non-safety, closed-loop system which provides cooling water for turbine auxiliaries and the containment building cooling and ventilating units. The heat sink for the system is normally circulating water, but the saltwater cooling or screen - wash pumps can be manually aligned as backups. The systs. consists of a turbine plant cooling water tank, two pumps, two heat , exchangers and associated valves, piping and instrumentation. The normal flow-path in the closed-loop system is from the turbine plant cooling water tank, through one of the pumps and its cooler to the supply header. Cooling water is distributed to the various loads and is returned to the tank. Some of the loads include circulating water pump motor bearing coolers, steam and feedwater sample coolers, turbine lube oil coolers, instrument and service air compressors, gen-erator hydrogen coolers, and containment building cooling and ventilating units. The turbine plant cooling water pumps are powered by 350-hp electric motors supplied from 4kV busses 1C and 2C. System piping to the containment building 4-6
cooling and ventilating units, between isolation valves CV 515 and 516, is safety related. If both TPCW pumps are off, it is possible for elevated por-tions of the TPCW system inside the containment building to drain. To minimize the potential for classical water hammer in the building cooling and ventilating units on restoration of flow, the TPCW flowpath to the building is isolated by the operator prior to starting a TPCW pump. The manual isolation valve is then throttled open slowly to restore building cooling. 4.9 Chemical and Volume Control System The chemical and volume control system (CVCS), shown on Figure 4.8, is a - safety-related auxiliary system with both normal operating and post-accident functions. Normal operating functions include purifying the reactor coolant system (RCS); maintaining proper water inventory in the RCS; providing seal - water for the reactor coolant pump seals; adjusting RCS boron concentration; maintaining the proper concentration of corrosion-inhibiting chemicals in the RCS; and filling and pressure-testing the RCS. A safety injection signal starts the preferred charging pump to provide borated water from the refueling water storage tank (RWST) for conditions when RCS pressure remains greater than safety injection system discharge pressure. The basic CVCS flowpath is from RCS loop A cold leg, through the regenerative heat exchanger and letdown orifices, to the residual heat removal (RHR) heat exchangers. The cooled and depressurized letdown flow then leaves the contain-ment building when it is purified by demineralizers before being sprayed into the hydrogen gas space of the volume control tank (VCT). RHR temperature re-corder TR600 indicates letdown temperature in the control room. The charging pumps take suction on the VCT and discharge through a flow control valve to the tube side of the regenerative heat exchanger and then to the loop A cold leg. Reactor coolant pump seal injection is from the discharge of the charging pumps through seal injection filters and individual flow control valves to the seals of the three reactor coolant pumps. Makeup of demineralized water and/or boric acid is supplied to the charging pump suction. The two centrifugal charging pumps have a capacity of 136 gpm at 2300 psig. The pumps are operated from the control room and are powered from 4kV buses 1C and 2C. Some CVCS components are provided with interlocks or automatic actuations based on plant conditions:
- 1. Letdown isolation valve LCV 1112 shuts if pressurizer level decreases to 10 percent but opens on restoration of level or loss of power.
- 2. Orifice block valves CV 202, 203 and 204 are closed by a safety injection signal (SIS) or loss of power.
- 3. Pressure control valve PCV 1105 fails shut on loss of power.
- 4. Charging pump suction transfers frem the VCT to the RWST on SIS or VCT low level; for a VCT low level transfer, suction is automatically returned to the VCT when level is restored.
- 5. If a pump is running, the nonrunning charging pump starts automatically on low charging header pressure or overcurrent trip of the running 4-7
pump. Pumps which trip on loss of power do not automatically restart when power is restored.
- 6. Safety injection starts the preferred charging pump and opens charging flow control valve FCV 1112.
Control power for CVCS letdown components is supplied from the utility and vital buses as follows:
- 1. 120 VAC utility bus -
LCV 1112, CV 202, 203 and 204 -
. Vital bus 1 - CV 525
- 3. Vital bus 4 - PCV 1105 -
- 4. Vital bus 5 - CV 526 4.10 Residual Heat Removal System .
The residual heat removal system (RHR) removes reactor decay heat and reactor coolant system (RCS) sensible heat during operations below 350 F. During operations at normal RCS pressure and temperature, the RHR heat exchangers are used as part of the chemical and volume control system (CVCS) to cool RCS letdown prior to purification. RHR system piping is also part of the hot leg injection flowpath during long-term recirculation cooling following a major loss of coolant accident. fhe RHR system shown in Figure 4.9 co'nsists of two pumps and two heat exchangers located inside the containment building. Flow during cooldown and refueling operations comes from a loop hot leg, through the RHR pumps and heat exchangers, to a loop cold leg. The low temperature and pressure RHR system is normally isolated from the RCS by a series of motor-operated isolation valves (MOV 813, 814, 833 and 834). An interlock is provided from pressurizer pressure instrumentation that permits opening valves 813 and 834 when pressure is less than about 370 psig. The interlock clears if pressure increases to about 400 psig, preventing the valves from being opened. Cooling water for the RHR heat exchangers comes from the component cooling water system (CCW). To provide a hot leg injection flowpath, for use during long-term recirculation cooling, RHR pump "A" manual bypass valve RHR 004 is normally left open. The recirculation flowpath enters the RHR system at the outlet of the RHR heat exchangers. Flow is then in the reverse direction through the heat exchangers to the bypass around pump "A" (RHR 004) and into the hot leg through MOVs 813 and 814. Prior to initiating normal RHR cooling, operators enter the containment building to close RHR-004. 4.11 dafety Injection System The safety injection system is designed to mitigate the consequences of a . loss of coolant accident. It injects borated water for initial core cooling and subsequently recirculates and cools spilled reactor coolant and injection water from the containment building sump for long-term cooling. The safety injection flowpath, shown on Figure 4-10, is from the refueling water storage tank (RWST) to the safety injection pumps, main feedwater pumps and the safety injection pentration to all three loop cold legs. An additional flowpath is from the RWST to the preferred charging pump and through the normal charging path. 4-8
4 1 daiety injection is automatically actuated by a signal from either low pres-surizer pressure (2/3) or high containment building pressure (2/3); it may also be initiated manually. Upon actuation, the main feedwater pumps are tripped and isolated from the condensate and feldwater systems and are realigned to take suction from the safety injection pumps, which provide borated water from the refueling water storage tank. Feedwater pump discharges are realigned to the safety injection header, the three cold leg injection valves open, and the feedwater pumps are restarted. The feedwater pump minimum flow is automatically transferred to the RWST by a safety injection signal to provide pump protection in case RCS pressure remains above the 1175 psig shutoff head of the pumps. The main feedwater pumps are 3500-hp motor-driven pumps capable of providing 10,500 gpm flow in their safety injection alignment. Power to the feedwater pumps is from 4kV buses 1C and 2C.
' To improve the reli sility of the feedwater pump realignment, valves HV 851 A and B and HV 853 A and B are equipped with bonnet vent valves. Tnese vent valves were added to vent the pressure between the double disks to prevent disk binding and impaired movement. Additionally, a small hole was drilled in the disks of main feedwater pump discharge check valves FWS 438 and 439 to prevent the formation of a hydraulic lock and improve the reliability of HV 851 A and B.
Because RCS pressure may remain above feedwater pump discharge pressure under some conditions, a safety injection automatically realigns the CVCS charging system to provide an immediate injection of barated water. Charging pump suc-tions are realigned to the RWST and the preferred charging pump starts to pro-vide flow through charging flow control valve FCV 1112. The operator may man-ually align the charging pump flow to the cold leg injection penetrations. During normal cooldown operations when RCS pressure is less than 500 psig, Technical Specifications require establishing two positive barriers between the RCS and the feedwater and condensate systems to prevent the flow of unborated water into the RCS. These barriers may be motor-operated valves, when closed ' and tagged with power removed; pneumatic-hydraulic valves, when closed and tagged with the respective hydraulic block valves closed; or manually operated valves, when locked closed or tagged. 4.12 Electrical Distribution System 4.12.1 General Description The SONGS Unit 1 electrical distribution system consists of the main transformer, auxiliary transformers A, B and C (which interface with the 220kV switchyard and the inplant electrical system), and the inplant electrical system (composed
, of the main and emergency diesel generators, the 4160V, 480V and 120V ac buses and loads, and the 125V dc system). Figure 4.11 shows the arrangement of the 220kV switchyard circuit breakers and buses for Unit 1. Figure 4.12 shows the arrangement of the main generator, auxiliary transformers, main transformer, 4160V system and 480V system. Figure 4.13 indicates the arrangement of 120V ac buses and the 125V dc system.
The main generator is a 500MVA unit that supplies its output power at 18kV to the main transformer and auxiliary transformers A and B. The main transformer is a 485MVA unit which supplies the main generator output to the San Diego Gas 4-9
and Electric (SDG&E) and Southern California Edison (SCE) power transmission systems through the 220kV SONGS switchyard. The main transformer steps up the 18kV generator voltage to the switchyard 220kV. Auxiliary transformer A and auxiliary transformer 8 are each rated 10/12.5MVA (OA/FA) and step down the 18kV generator voltage to supply inplant 4160V buses 1A and 1B, respectively. The main and auxiliary transformers are directly connected to the generator bus. The main transformer output is connected to the northeast and northwest 220kV switchyard buses through circuit breakers (CB) 4012 and 5012. Auxiliary transformer A supplies 4160V bus 1A through circuit breaker 11A04, and auxiliary transformer 8 supplies bus 1B through CB 11B04. The main generator 18kV bus has a no-load motor-operated disconnect (MOD) switch, - which can be used to isolate the generator from the transformers, and which allows buses 1A and 18 to be backfed from the switchyard through the main and auxiliary transformers. (The latter feature is provided to meet the delayed- - access circuit requirement of General Design Criterion (GDC) 17.) Auxiliary transformer C is the immediate access circuit (required by GDC 17) between the switchyard and the in plant safety-related electric distribution system. Auxiliary transformer C is a 50MVA unit with one primary winding (designated as H winding) and two secondary windings (designated as X and Y windings). The H windings of auxiliary transformer C are connected to the switchyard through circuit breakers 4032 and 6032 and step down the 220kV from the switchyard to 4160V. The 4160V X winding supplies bus 1C through circuit breaker 11C02, and 4160V Y winding supplies bus 2C through 12C02. A current-limiting reactor and a bypass breaker are provided on each of the X and Y wind-ing outputs. The reactors are used only when the associated diesel generator is being load tested; they are used to reduce any current associated with a fault during this mode of operation. Tie breaker 11C01 is provided between bus IA and bus IC and tie breaker 12C01 between bus 18 and bus 2C. The two emergency diesel generators, each rated at 6000kW, are designed to sup-ply 4160V buses 1C and 2C during emergency conditions when offsite power is unavailable to the buses. The 4160V buses 1C and 2C supply the unit's 480V switchgears 1, 2 and 3 through station service transformers. These switchgears in turn supply nine motor control centers (MCC's 1, 1A, 18, IC, 2, 2A, 28, 3, 3A) and all 480V loads. One set of such loads includes the battery chargers that are associated with 125V dc buses 1 and 2. Unit 1 has seven 120V ac vital buses and one 120V ac utility bus which supply power to various instrumentation and control systems. Vital buses 1, 2, 3, 3A, 5, and 6 are normally supplied by inverters off the 125V dc buses. Vital bus - 4 and the utility bus are supplied by the unit's 480V system (MCC 2 is the normal supply and MCC ? is the alternate) through single phase transformers. 4.12.2 220kV System During normal plant operation, circuit breakers 4012 and 6012, associated with the main transformer, and circuit breakers 4032 and 6032, associated with auxiliary transformer C, are closed. These breakers can be operated (i.e., closed or opened) from the control room by control switches located on the vertical panel. These breakers are automatically tripped by protective relays associated with the switchyard and with their associated transformers. One such protective relay is the differential relay associated with auxiliary 4-10 i
transformer C, which on actuation will trip open circuit breakers 4032 and 6032. Similarly, circuit breakers 4012 and 6012 open automatically upon actuation of 1 protective relays associated with the main generator, main transformer, and auxiliary transformers A and 8. Actuation of the unit trip or undervoltage on ; both buses 1C and 2C trip open breakers 4012 and 6012. Actuation of the above-mentioned protective trip relays associated with the main and auxiliary trans-formers and the main generator would also actuate the generator lock-up and associated alarm buses. 4.12.2.1 Closing '20kV Circuit Breakers for Backfeeding
~ Circuit breakers 4012 and 6012 are associated with generator output to the 220kV switchyard. Normally these breakers are closed to supply generator output to the switchyard. However, during situations when auxiliary transformer C is
- unavailable and the unit is tripped, the only available path for offsite power to supply the station distribution system is through the main and A and B auxil-iary transformers. In this mode of operation, the 18kV bus must be isolated from the main generator. This isolation and alignment of the distribution system are automatically performed by the " loss-of-voltage auto transfer" scheme. At the completion of the automatic transfer scheme, operators are required to man-ually close breakers 4012 and 6012 to complete the backfeed operation. In order to close the breaker manually, the operator must:
- 1. Depress the unit trip reset pushbutton in order to remove all trips on the breakers.
- 2. Insert the synch. selector switch in the appropriate control location and turn it on.
- 3. Depress the synch. check bypass pushbutton.
- 4. Turn the circuit breaker control switch to the close position.
These four steps are required to close the first switchyard circuit breaker, normally CB 4012 associated with the 220kV northeast bus. To close the second breaker, CB 6012, only steps 2 and 4 are required. Step 1 is no longer neces-sary since unit trip reset has reset the generator lock-up, and step 3 should not be used in order to allow the synch. check relay 25X to perform its function of verifying synchronism between the northeast and northwest 220kV buses. (When the main generator is operating, synchronizing check relay 25X provides a per-missive to assure that the generator output is within limits with respect to the running 220kV system.) 4.12.3 4160-Volt System i = Bus IA is normally supplied by auxiliary transformer A through breaker 11A04. It can also be tied to bus 1C through tie breaker 11C01 and be supplied by auxiliary transformer C during startup or whenever needed. The loads on this bus are the motors for reactor coolant. pumps A and C. Auxiliary transformer B normally supplies Bus 18 through breaker 11B04. Its alternate supply path is from auxiliary transformer C to bus 2C and tie breaker i 12C01. The loads connected to this bus are the reactor coolant pump 8 motor, and the main and spare exciters of the main generator. Bus IC, a safety-related bus, is normally supplied by the X winding of auxiliary transformer C via circuit breaker 11C02. It can be supplied by auxiliary transformer A through bus 1A and tie breaker 11C01. Emergency diesel 4-11
generator 1 can also supply this bus through its associated breaker 11C14. The following loads are connected to bus 1C.
- 1. West feedwater pump motor
- 2. North circulating water pump motor
- 3. West safety injection pump motor
- 4. Southwest condensate pump motor
- 5. Charging pump B motor
- 6. Northwest condensate pump motor
- 7. West heater drain pump motor
- 8. Normal ~ lighting transformer feeder
- 9. Station service transformer (SST) 1 feeder -
- 10. Station. service transformer 3 feeder
- 11. South turbine plant cooling water (TPCW) pump motor Bus 2C is the other safety-related 4160V bus which is normally supplied from the Y winding of aux liary transformer C through breaker 12C02. It can be i
supplied through tie breaker 12C01 from bus 18. Its associated emergency diesel generator 2 can also supply the bus through breaker 12C15. The loads connected to this bus are as follows.
- 1. East feedwater pump motor
- 2. South circulating water pump motor
- 3. East safety injection pump motor
- 4. Southeast condensate pump motor
- 5. Charging pump A motor
- 6. Northeast condensate pump motor
- 7. East heater drain pump motor
- 8. North TPCW pump motor
- 9. Station service transformer 2 feeder
- 10. Standby lighting transformer feeder
- 11. Switchyard feeder
- 12. Alternate feeder to SST 3 Tie breakers 11C01 and 12C01 can be opened and closed from the control room.
They can also close automatically on station loss of voltage auto transfer sequence. They trip and lockout on safeguard load sequencing system (SLSS) operation. (The SLSS actuates and sequences the various emergency safeguard features in the event of a safety injection signal (SIS), loss of offsite power (LGP), loss of 4160V bus 1C/2C (LOB), or SIS and LOP.) The tie circuit breakers are provided~with overcurrent protection. The source breakers to buses 1A and IB (11A04 and 11804) can also be operated , from the control room. They too automatically close during the station loss of ! voltage auto transfer scheme. These circuit breakors trip on overcurrent . protection actuation, on actuation of protective features provided for the associated transformers and generator, and on unit trip. Source breakers 11C02 and 12C02 for buses 1C and 2C can be opened and closed from the control room. These circuit breakers do not have an auto close feature and trip on overcurrent, auxiliary transformer C protection (such as differential relay) and during the station loss-of-voltage auto transfer scheme. i l 4-12 l I i
Diesel generator breakers 11C14 and 12C15 can be closed from the control room during a loss of power (LOP), provided the diesel generator is at valid voltage and frequency, the respective bus 1A-1C or bus 18-2C tie breaker and auxiliary transformer reactor bypass breaker are open, and the initiating LOP signal has been reset at the SLSS surveillance panel. Buses IA and 18 are provided with undervoltage alarms and undervoltage relays whose contacts are used in the trip circuits of the reactor coolant pump motors. Buses IC and 2C also have undervoltage alarms and relays. Undervoltage relay actuation on buses IC and 2C will trip certain connected loads, initiate the loss of voltage auto transfer scheme, and start the associated emergency diesel
. generators. Buses IC and 2C are each provided with a ground detector relay, which alarms on a 10 percent ground (the set point is 21 volts), and a dual scale (10 percent and 100 percent) ground voltmeter located on the control room electrical board. Buses 1A and IB do not have ground detectors that directly monitor grounds on the bus, but have ground detector relays and voltmeters that monitor the secondary neutrals of auxiliary tranformers A and B. The alarm in this case is set at 6.9 volts.
4.12.4 480-Volt System The 480V system is supplied power from the 4160V system through station service transformers (SST). SST 1, connected to 4160V bus IC, supplies 480V bus 1; SST 2, connected to 4160V bus 2C, supplies 480V bus 2; and SST 3, normally connected to 4160V bus 1C with 4160V bus 2C as an alternate source, supplies 480V bus 3. SST 3 source breakers 11C11 and 12C11 are designed to be shed on undervoltage of 4160V bus IC and 2C, respectively. The following loads are connected to 480V bus 1. Auxiliary feedwater pump G10S Sphere Enclosure Building (SEB) normal ventilation fan A40 SEB normal exhaust fan A42 North screenwash pump North saltwater cooling pump East condenser vacuum pump North refueling pump North component cooling pump East RHR pump West fire pump East recirculation pump
. South instrument air compressor Pressurizer heater group A Pressurizer heater group C , Battery charger set A MCC's 1, 1A, 18, 1C This bus is provided with a ground detector system (a relay at the bus and e dual-scale voltmeter in the control room). The bus voltage is indicated in the control room and undervoltage relays are also provided. Bus undervoltage actuation trips all feeder breakers, except the MCC feeders, instrument air compressor and battery charger.
4-13
The loads on 480V bus 2 are : East fire pump South screenwash pump West condenser vacuum pump Component cooling pump South refueling pump West RHR pump South saltwater pump SEB vent supply fan A41 - SEB exhaust fan A43 Instrument air compressor - West recirculation purrp Pressurizer heaters groups B and D Battery charger set B - MCC's 2, 2A, 28 The ground detection and bus voltage monitoring for this bus are similar to those of bus 1. Also similar is that undervoltage on the bus will trip all feeder breakers except MCC feeders, instrument air compressor and battery charger. The foiiowing loads are connected to 480V bus 3. Turbine auxiliary lube oil pump Auxiliary saltwater cooling pump North instrument air compressor Auxiliary cooling pump Boric acid batching heaters South component cooling pump Switchgear room HVAC distribution panel MCC's 3, 3A Ground detection and voltage monitoring, similar to those on the other 480V buses, are provided. On a bus undervoltage condition the breakers trip loads from the bus for the cooling water pump, turbine auxiliary lube oil pump, auxiliary cooling pump, boric acid batch heaters, and auxiliary saltwater cooling pump. 4.12.5 120-Volt AC System This system is designed to provide a reliable, regulated and redundant source - of single phase 120V ac pcwer for plant controls and instrumentation. As shown on Figure 4.13, it consists of inverters, transformers, regulators, automatic l transfer switches, seven vital buses, and one utility bus. . Inverters 1, 2 and 3, connected to 125V dc bus 1, are the normal source of power to vital buses 1, 2, 3 and 3A. These buses transfer automatically to the 37.5kVA transformer if the inverter voltage or frequency deviates from pre-set limits. Similarly, vital buses 5 and 6 have inverter 5 as the normal supply with the 7.5kVA transformer as back-up. Vital bus 4 is normally supplied from the regu-lated 7.5kVA supply, with automatic transfer to the 37.5kVA back-up supply. Utility bus transfers from the 37.5kVA transformer to lighting switchboard back-up supply. The two single phase transformers which supply power to vital 4-14
buses 1, 2, 3, 3A, 4, and utility bus, are supplied 480V from MCC 1 and MCC 2 through a manual master transfer switch (the normal supply being MCC 2). 4.13 Safeguard Load Sequencing System (SLSS) The primary function of the safeguard load sequencing system (SLSS) at SONGS 1 is to detect and react to low pressurizer pressure, high containment building pressure, and 4160V bus 1C and/or 2C undervoltage signals. The SLSS actuates and sequences emergency safeguard features (E56 in the event of a safety in-jection signal (SIS), loss of offsite power (!N), loss of 4160V bus 1C or 2C (LOB), or a simultaneous SIS and LOP (SISLOP). The SLSS is composed of two independent and redundant sequencer trains desig-nated as sequencer 1 and sequencer 2. Each sequencer is composed of one logic ~ cabinet, one termination cabinet with two cable assemblies, and one remote sur-veillance panel. Tne sequencers receive power from the 125V dc system. Each sequencer logic uses a two-aut of-three low pressurizer pressure or high containment pressure scheme to initiate an SIS. The LOP and LOB logic is a one-out-of-two bus undervoltage arrangment. Each sequencer has redundant sub-channel X and subchannel Y with inputs from pressurizer pressure, containment pressure, and 4160V buses IC and 2C undervoltage relays. Both subchannels must actuate in order for the sequencer to actuate. The two remote surveillance panels, one for each sequencer, are mounted in the control room on their associated diesel generator control board. Each panel contains the SIS manual initiation and reset pushbuttons and switch, the LOP manual initiation and reset pushbuttons and switch and eight status lamps. Six of the eight lamps are located on the right-hand side of the panel and they give the status of SLSS load group A through F. These lamps are normally il-luminated and go off when their respective load group is sequenced on. One of the remaining two lamps is a normally energized lamp indicating availability of the sequencer power supply. The final lamp is a "dcor closed" indicator that will extinguish if any one of the four sequencer doors is open. The load groups are associated with the load sequencing that is automatically initiated by the SLSS upon a SISLOP signal. The load groups are designated as follows: Load Group A - Time 0 seconds. This load group is designed to perform its function immediately with no timing circuitry involvement. (There is a 10-second allowance for the diesel generators to reach rated voltage and frequency.) o Load Group B - Time 11 seconds. This load group is designed to load onto the safety-related buses 1 second after the diesel generator output breaker closes. Load Group C - Time 12 seconds. This load group is designed to load 2 seconds after diesel generator breaker closure. Load Grop D - Time 21 seconds. This load group is designed to be energized 11 seconds after the diesel generator output breaker is closed. 4-15
Load Groups E and F are spares and not used. The functioning of the SLSS and its load groups for various conditions are as follows:
- 1. SIS initiation with offsite power available will cause
- a. a reactor trip and a unit trip,
- b. the start of both diesel generators, and
- c. acutation of all SI loads in all the load groups through the sequencer without any timing sequence.
- 2. SIS initiation with loss of offsite power will cause
- a. a reactor and unit trip, both diesel generators to start, b.
- c. load stripping from 4160-V and 480-V buses, and, d; the diesel generator output breakers to close and initiate load sequencing of load groups A through D.
- 3. Loss of offsite power, LOP, will cause
- a. a reactor trip and a unit trip, and
- b. both diesel generators to start.
- 4. Loss of 4160V bus 1C or 2C, LOB, will start the associated diesel generator.
1 When conditions 1 and 2 above occur, the load group A through D status lamps at the remote surveillance panels will be off, indicating that the load groups have sequenced on. Under condition 3, only the group A status lamp at each panel will extinguish. Under condition 4, only the group A lamp on the panel associated with the lost bus will go off. 4.14 Diesel Generators The two redundant diesel generators provide a reliable standby source of power to safety-related equipment if offsito sources are not available. Either diesel generator can supply sufficient safety-related loads to respond to accident conditions or place and maintain the reactor in a safe shutdown condition. The diesels are 8375-hp, turbo-charged units supplied by Transamerica-Delaval. Each 4KV generator has a continuous rating of 6000kW, with a ten percent overload rating for 2 hours. The generators start automatically on a safety , injection signal or loss of power signal (LOP). However, the generators are not automatically connected to their respective buses and loaded sequentially unless both safety injection and loss of power conditions exist. Fuel oil for the generators is stored in two 50,000 gallon tanks. Each gener-ator also has a day tank containing a Technical Specification minimum of 290 gallons for a 45-minute supply at full load. A low day tank level starts an ac powered fuel oil transfer pump. Fuel is supplied to the diesel injectors by an engine-driven pump, with a dc powered standby. Lubricating oil is 4-16
=. .-
provided by an engine-driven pump with an ac motor-driven backup. Each diesel has an oil cooler supplied by the diesel cooling water system. The dedicated cooling water system for each generator circulates coolant from the engine and its auxiliaries through a fan-cooled, water-to-air heat exchanger (i.e., a radiator). Coolant is circulated by an engine-driven pump and the radiator fans are driven by an ac motor. A diesel can operate unloaded, without overheating, for 39 minutes with no power to the radiator fan motor. The redundant air-starting system for each diesel consists of two compressors, two receivers and associated valves, piping and auxiliaries. The receivers are designed to provide five start attempts without recharging. Power to the ac powered diesel auxiliaries comes from diesel generator motor
~ control centers (MCC) 18 and 28. These MCCs ultimately receive power from 4kv buses 1C and 2C, which are normally powered from offsite, but receive standby power from the diesel generators.
4.15 Containment Building Isolation The purpose of the containment building isolation system is to minimize the release of radioactive materials to the ervironment by automatically isolating all nonessential systems that penetrate the building. Isolation is initiated by a containment building high pressure or safety injection signal. Main and mini purge ventilation isolation also occur if high containment building radio-activity is sensed by radiation monitor R-1212. Automatic isolation valves are provided in all normally open, nonessential systems that penetrate the contain-ment building. These valves fail shut on loss of power or operating air. Control power for the containment building isolation signal actuation logic and building isolation valves (except for main purge valves POV 9 and 10, and mini-purge valve CV 10) comes from either 125 VDC buses 1 and 2 or from inverter-supplied 120 VAC vital buses. Control power for POV 9 and 10 and CV 10 comes from 480 VAC bus 1. Containment building radiation monitor R-1212 receives control power from 120 VAC vital bus 4, which is not inverter-supplied. During power operations the containment mini purge flowpath through CV-10, 40 and 116 is normally open to maintain a slight negative atmospheric pressure inside the building. This flowpath is required because of air leakage from valves and instruments in the building. CV-10 fails shut on loss of power and remains shut after power is restored. CV-10, 40 and 116 receive an isolation signal when radiation monitor R-1212 alarms or loses control power. The radia-tion monitor must be reset before the mini purge isolation valves can be reopened. Normally shut containment building main purge valves POV 9 and 10 are isolated l , by locked-shut manual valves in series except during cold shutdown and refueling. 4.16 Station Personnel This section describes the personnel organizations at the San Onofre site irvolved in the November 21, 1985 event. 4.16.1 Administrative Organization The administrative organization of San Onofre is shown in Figure 4.14. Most of these departments are divided into those supporting Unit 1 and those who support 4-17
both Units 2 and 3. The site organization consists of approximately 1,500 permanent staff members and 1,000 contractor employees. They support the operation of all three units at the site. In maintenance, approximately 90 people work exclusively on Unit 1, 300 work on both Units 2 and 3, and 150 people divide their time among all three units. The plant Operations Depart-ment functions under routine conditions to operate all three units. Of the 300 plant operations personnel, approxiinately one-third are assigned to Unit 1. The on-shift organization is responsible for the immediate concerns involved in operating all three units. 4.16.2 On-Shift Organization The on-shift organization comprises several different groups working around-the-clock on the unit to produce electricity. Personnel from several other depart-ments, in addition to those from the operations staff, work on-shift. The on- - shift work assignments routinely come from administrative supervisors, but all activities that affect the unit are coordinated through the Shift Supervisor. The Shift Superintendent is responsible to the Operations Department Plant Superintendent at Unit I for safe operation of the plant under all conditions. To support this responsibility under routine conditions, the Shift Superinten-dent has authority over all personnel whose activities affect Unit 1. There are a minimum of six on shift operators, in accordance with plant technical specifications and site procedures, who report to the Shift Superintendent (see Figure 4.15). The Shift Superintendent interacts with other organizations in support of his work. The Shift Superintendent communicates with the Technical Department through the Shift Technical Advisor (STA). The STA provides technical informa-tion or recommendations to assist the operating crew. The Shift Superintendent can also initiate on-shift maintenance without going through the administrative organization by issuing a Shift Supervisor's Accelerated Maintenance (SSAM) order. Finally, the Shift Supervisor exercises authority by assuming the duties of Emergency Coordinator during the initial stages of emergencies. The Shift Superintendent holds an SRO license. The next senior position on shift is the Control Room Supervisor (CRS), who also holds an SR0 license and normally exercises the Control Room Command Function for the Shift Superintendent. This function is formally assigned to a single operator who is then responsible for exercising the necessary decision and command authority for overall activities or operations affecting either the safety of the general public, station personnel, or the plant. The Control - Room Supervisor is required to remain in the control room, unless relieved, to maintain a broad perspective on operations by not becoming too involved with any single operational activity, and to assume the authority and duties of the . Shift Superintendent in the Shift Superintendent's absence from the control room. The CRS is also expected to make a complete tour of the plant once during the shift. In an emergency, when the Shift Superintendent assumes the Control Room Command Function, the CRS becomes responsible for implementing the appro-priate procedures and for directing other operators. The Control Operator (CO), the next senior position in the chain of command, j holds a reactor operator (RO) license. The C0's primary responsibility is the ! 4-18
safe and efficient operation of assigned equipment. For Unit 1, the C0 is responsible for all plant equipment. This responsibility is carried out through direction of licensed and non-licensed operators at the unit. Although all the other personnel on duty are expected to make plant tours each shift at some time during the shift, the C0 is required to stay in the Control Room for the entire shift. For a plant trip or transient, the C0 takes control of the pri-mary and electrical control panels. The Assistant Control Operator (ACO) is responsible for carrying out the direc-tions of the CO, but must also assist in directing the activities of licensed and nonlicensed operators at the unit. The AC0 holds an R0 license and remains in the control room, except for the required plant tour. During a trip or tran-sient, the AC0 takes control of the secondary control panels, nonessential loads, turbine generator shutdown and Emergency Plant Implementation Procedure (EPIP)
, duties.
The remaining two members on shift are nonlicensed Nuclear Plant Equipment Operators (NPE0s), who take directions from the Control Operators and spend most of their time outside the control room making inspection tours and keeping logs on equipment operations. The NPE0s are responsible for keeping the Control Operators aware of plant conditions. In the event of a plant trip or transient, NPE0s are required to contact the control room for directions. The on-shift operators remain as a unit through shift rotations and training and consider themselves an operating team in running the plant. Five shifts are employed, with three on-shift around-the-clock, one in training, and one providing extra personnel to cover on-shift absences or additional support. On-shift personnel are also supported with extra personnel. The Shift Technical Advisor (STA), who is available 24 hours per day, holds a SR0 license. The STA reports to the control room if a trip or transient occurs and advises the Shift Superintendent. Operator trainees, in addition to the shift staff, often fulfill shift responsibilities under the supervision of the on-shift operator. Also, off-shift operators are available to provide support beyond that within the capability of the on-shift personnel. Thus, on-shift personnel have regularly scheduled assistance and as needed additional resources to support them. In emergency conditions, including the declaration of an Emergency Plan Action Level, the Shift Superintendent becomes the Emergency Coordinator until relieved by a senior administrative manager: the Plant Superintendent, Operations Manager, or Station Manager. The Emergency Coordinator is responsible for notifications, j - evaluations, protective actions, event classification, exposure control, and emergency response coordination. When activiated, all organizations have repre-sentatives working directly for the Emergency Coordinator to provide any
. necessary assistance.
4.16.3 Human Factors Affecting Operator Performance Several factors affect operator performance. These include the control room where they perform most of their actions, the training the operators receive, and the procedures that they follow. These subjects are addressed in the fol-lowing sections. 4-19 J
4.16.3.1 Control Room The Unit I control room is a relatively conventional design with center J-shaped console surrounded on three walls by vertical panels. The back of the control room has desks for operators and doors to the offices of the shift superintendent and the nuclear operators assistant. The back wall of the control room has a glass window for viewing the control room from the adjacent Technical Support Center. Figure 4.16 shows the control room from the Shift Superintendent's office door and Figure 4.17 is the floor plan for the control room and adjacent offices showing major panels and controls significant to this event. The con- - trol room has not undergone any significant upgrades required by NRC's TMI Action Plan. The site's detailed control room design review plans to upgrade the con- - trol room were scheduled for submittal to the NRC in December 1985. The panels showing indications that played a significant role in the event are - shown in the section addressing personnel performance, Section 7, Figures 7.2, 7.3, and 7.4. Figure 7.1 shows the orientation of these specific panels within the control room. For a detailed discussion of control room indications and data recording systems, see Section 4.17. 4.16.3.2 Training Qualification at each position of the shift crew includes completion of specific individual training and on-the-job experience. Several positions also require classroom training. The NRC's R0 and SR0 licensing programs are included in the licensee's training program, but are supplemented by additional training and experience. Operators are trained for a week annually on the simulator at Zion Nuclear Power Station. Although the Zion simulator is not identical to the Unit 1 control room, it is reprogrammed to model Unit 1 characteristics and behavior for this training. The significant advantage of simulator training is that each shift trains as a team and practices the teamwork essential to responding i to trips and transients. An STA is included in the shift's simulator training, while trainees or other personnel are not. 4.16.3.3 Procedures The emergency operating procedures at Unit I are based on the NRC-approved ge-neric Westinghouse Owner's Group technical guidelines. Since the TMI accident, a significant shift in the philosophy of operations during emergencies has oc-curred in the nuclear industry. Previously, procedures were written based on - two assumptions: (1) that one event would cause an emergency and, (2) that operators could quickly diagnose the emergency's cause. TMI demonstrated viv-idly that both assumptions could be wrong. Since then, the industry has concen- . trated on writing procedures to focus on maintaining functions necessary to plant safety, rather than those focusing on specific events. This new function or symptom orientation, in its pure form, is not viewed as .ifficient for address-itg well-understood minor upset conditions. Therefore, hybrid approaches have been developed, a.nong which is the Westinghouse Owners Group (WOG) technical guidelines. The Unit 1 procedures employ the WOG approach of attempting to respond to specific events with " optimal recovery guidelines" while requiring that checks be made on the status of a set of " critical safety functions." 4-20
i The emergency procedures, beginning with " Reactor Trip or Safety Injection" procedure 501-1.0-10, are implemented when the reactor trips or should have tripped, when a safety injection occurs, or if there is a loss of electrical power. The procedure requires operators to verify proper system trips for the reactor and turbine, to check electrical power available, and to check SI. If these checks identify abnormalities with the reactor trip or electrical power, the operator is then directed to use the emergency operating instructions (E0Is) for anticipated transients without scram or loss of all AC power, as appropriate. If SI actuated, the procedures direct operators to ensure the line up of neces-sary systems and then perform diagnostic checks and plant stabilization until SI can be terminated. If SI did not actuate, the operators are directed to . follow the routine trip response procedure. The " Reactor Trip Response E0I," procedure 501-1.0-11, provides guidance on achieving initial and then long-term plant stability. Simultaneous with the use of these procedures, operators evaluate the Critical Safety Functions Status Trees (procedure 501-1.0-1). The procedure sets and directs priorities for subcriticality, core cooling, heat sink availability, reactor coolant system integrity, containment building integrity, and reactor coolant system inventory. Depending on the status and priority of these, operators are directed to procedures designed to restore the necessary critical safety functions. The evaluation of the critical function status trees is typically performed by the STA, while the operators are performing ti;e other E0Is. The STA periodically reports the status trees status to the Shift Superintendent as an independent check of the safety condition and succass of operator actions. 4.17 Control Room Indication The majority of the plant's instrumentation and control power supply require-ments are provided by the 120V ac buses. Many indicators, recorders and meters in the control room are dependent on these buses. Two of these buses, vital bus 4 and the utility bus, are supplied by the plant ac distribution system and can lose power under total loss of ac power conditions. On loss of power to these buses, the control room indicators, recorders and other instru-mentation that are dependent on these buses, fail or adjust into different states. A detailed list of the control room components affected and the state of their failure are provided below. 4.17.1 Loss of Vital Bus 4 The following is the list of indicators and recorders which are supplied power from vital bus 4 and their status upon loss of bus power. Control room instruments off this bus are identified by a green dot on the instrument. Status After Instrument Loss of Power Indicatc;s: Vital bas 4 voltaga indication light off Analog rod position indication system off Pressurizer pressure indicator PI 434 fails low RCP seal water return temperature indicators fails low TI 1116 A, B & C 4-21
i Letdown TI 1103 B fails low Letdown flow FI 1104 fails low Charging flow FI 1112 fails low Charging line temperature TI 1103 A fails low ARMS channels R 1231, R 1232, R 1233, alarms R 1234, R 1235, R 1236 & R 1237 ORMS channels R 1211, R 1212, R 1214, alarms R 1215, R 1216, R 1217 & R 1218 ERMS channels R 1250, R 1251, & R 1252 alarms Stack iodine monitoring channels R 1219, alarms R 1220 & R 1221 RCP seal water return cross-tie to off; valves - VCT valves CV 410 & 411 indications fail closed Recorders: . Pressurizer level LR 430 fails low Pressurizer pressure (narrow range) PR 430 fails low Pressurizer pressure (wide range) PR 425 fails low Pressurizer liquid and vapor TR 430 fails low VCT 1evel LR 1100 fails low RCS loop Tave TR 401 fails low RCS cold leg TR 402 chart drive stops; indication good Tave/ Tref TR 405 fails low SGs steam flow, feed flow, level fails as-is YR 456, 457 & 458 Steam and feedwater pressure R 8 chart drive stops; indication good Condenser backpressure PR 480 fails low Radiation monitor RLR 1200 fails low RCPs bearing temp TRC 446 fails to print / advance Charging and SI pump bearing temp fails to print / advance TRC 1119 Component cooling water heat exchanger chart drive stops; outlet temp TR 606 indication good 4.17.2 Loss of Utility Bus The following is the list of indicators and recorders which are supplied power from the utility bus and their status upon loss of bus power. Control room - instruments off this bus are identified by a yellow dot on the instrument. Status After Instrument Loss of Power . Indicators: Utility bus voltage indicator light off Pressurizer power relief line temp fails low TIC 433A Pressurizer safety line temp fails low TIC 433 8 & C
- Pressurizer spray valves PCV 430C&H off; valve indicating lights functional 4-22
RCP seal water return valves off; valves PCV 1115 A, 8 & C indicating lights fail open RCP seal water valve CV 291 off; valve indicating lights fails open RCPs seal bypass valve CV 276 fails; valve indicating lights fails closed Letdown control valve LCV 1112 off; valve indicating lights fails open Letdown orifice isolation valves off; valves CV 202, 203 & 204 indications fail closed Charging isolation valves off; valves " CV 304 & 305 indications fail closed Excess letdown isolation valve offs; valve CV 287 indication light fails closed Pressurizer relief tank vent valves off; valves CV 542 & 543 indications fail closed Steam dump to condenser valves off; valves CV 3 & 4 indications functional Recorders: RCP seal leakoff FR 1117 and 1118 fails low RCP seal water supply temp chart drive stops; TR 1111 indication continues Turbine metal temperatures fails to print / advance R 2 and R 3 Misc bearing temperatures fails to print / advance R 4 and R 4A Generator and transfermer temp fails to print / advance R5 Generator gas and exciter temp fails to print / advance R6 Feedwater cump suction pressure temp. fails as is; and temperature R7 press. fails high ' Average feedwater temp fails as is; chart TR 456 drive fails Condensate flow FR 480 fails low RCP vibration detectors fails low ' 4.17.3 Data Recording Systems . The function of data and event recording is provided by control room data re-corders, a control room event recorder, a critical function monitoring system, and the oscillograph system. All of these, except the oscillograph system, are
- routinely used for diagnosing the causes of unscheduled reactor trips and to ensure that safety-related equipment functioned properly. The oscillograph system records transients on the electrical distribution system immediately following 3 system disturbance and provided valuable information on this event.
The control room chart recorders at Unit I are the common analog recording devices for plotting individual or small groups of parameters against time. Several types of chart recorders are used with different power supplies for the chart drive and pens. Some recorders are powered from uninterruptible inverter-powered 120 VAC vital buses, while the other recorders are not. 4-23
The control room event recorder is a 40-track discrete event recorder that accurately records the trip and reset of 38 t: bistables associated with the reactor and turbine generator. Of the other two tracks, one indicates the recorder's speed, while the second is a spare. The power supplies for the event recorder are from safety related sources. The paper drive for the recorder normally operates at 3/4 inches per hour, but on a trip it switches to a speed that corresponds to 3/4 inches per second. The event recorder was replaced in 1984 after 14 years of service. The critical function monitoring system consists of a small general purpose computer, the FOX III, providing on-line display of selected plant parameters in different formats; it has a xenon prediction capability. Its periodic - recording of plant parameters is also available for review after any trip. The system has hard-copy printers and CRT displays. One CRT, located near the window in the Technical Support Center (TSC), is visible from the control room. - The system's computer and all but one of the printers are located in the TSC adjacent to the control room. The other printer is located in the Emergency Operations Facility (EOF). Under normal conditions, the system records the current value of all parameters each minute and maintains this information for 24 hours. Upon sensing a trip, the system records another 5 minutes and then automatically prints out data for the 25 minutes prior to the trip and 5 minutes following. The system can also track and print specific parameters on user request at 1-second intervals. The normal power supply for the system is the MCC IC bus powered from the 4kV IC bus; the central processor has an internal battery backup to perserve main memory for short power outages. To support analysis of electrical transients, the oscillograph system records 29 data points of the electrical system. The system is not normally considered part of the plant's event recording systems because it primarily deals with the output of the reactor plant's turbine generator. This system records phase-to-phase and neutral voltages for the 4kV buses and the turbine generator, as well as the position of the control valve, stop valve, and reactor scram breakers. The response time of the system is designed to record electrical faults and, therefore, the system is fast enough that the tracings show actual voltage sine waves associated with the 60-hertz bus frequency. The oscillograph system is not normally recording but when one of ten initiators occurs, the system records its sensor data in sequence. The trips are (1-5) undervoltage setpoints on the 4kV buses and the generator, (6) over voltage at the generator, (7) generator frequency out of normal band, (8) generator stator ground, i9) main transformer neutral current, and (10) a manual control switch. The deiice records for a fixed time, then calibrates all channels with a standard rignal, and resets for another initiation. - 4.18 NRC Emergency Response Organization When immediate NRC notification is required, the first person called at NRC is the Headquarters Duty Officer. Several state and local organizations are also kept informed of events at the plant. Because these agencies usually require notification prior to the NRC, the NRC's notification may be up to 1 hour or 4 hours after an event, depending on the plant's classification of the event. Calls are usually made to the NRC using the dedicated-line Emergency Notifica-tion System (ENS). Phones are installed at Unit 1 in the control room, the Shift Superintendent's office and two nearby offices (see Figure 4.17). 4-24
l The NRC Duty Officer serves as the initial NRC notification channel to the plant for minor events through emergencies. The Headquarters Duty Officers stand 12-hour shifts and control the ENS. They do not have detailed knowledge l of each plant reactor but have general training on each reactor type. The initial notification is passed by the Headquarters Duty Officer to the appropriate NRC Regional Duty Officer and the region initially has the lead responsibility for the NRC's response. The Regional Duty Officer notifies the appropriate regional official. The level of response is based in part on the plant's classification of the event in accor-dance with their NRC-approved and tested emergency plan. Based on an evaluation of plant conditions and adequacy of the licensee response, the Regional Adminis-trator, in consultation with the Headquarters Emergency Officer, recommends actions necessary to place NRC in the appropriate response mode. The region . has the responsibility to activate NRC's formal response organizations when a licensee declares an " Alert" by placing NRC in " standby" status. This response level activates the regional and headquarters incident response centers but the region retains the lead responsibility. Further levels of response are imple-mented based on guidelines described in NUREG-0845, " Agency Procedures for the NRC Incident Response Plan." O J 4-25
E n e= a n ls.i a n% 8 E i ~ m :a m :sll L 7 b ,
'k E 'h b I
- - I l..i b
n l.i m b I Jo- li 14 l 33 = I 5 ll1 23! h a a M ih 5 . 5 Xi b " XI XI h* E3 ! b b [ e-I . 1
~
T ____ T ___ g
=C C I -4 - -
C- - - I
. g.g .==
GG IIR I E D* T A If U AH L C I UtS CO QE L O T* R IR CE T
" OA a T W
= -'
K N . nq A 'L 7 T 3
'a N
W O D v W n O L 8 w o C d X F w l o - B r _ X t o _. s a r Wm r e _ 8 0 A G n e 8mf L HE C s21 GV IE HL 6 1 V C
?
r
% w5 r m 1N1 0O2ir 1 ' a 1 e ETIR SA OLO
=L S t
L CA R 2 GL IL NA Tr C
/ s61 4
e AT LU F w5 r r UO u C RRE g i OT F O A TW es' .
' ' ,l
. e bgu h=Efs EB R R R O O O N T N T N T W WDI O NO N W S
R O O O W O AAR WE N OE O W OmO OY0i x T 0 A R E K O X rat OsI T AA I D x L S O A N E T A T A G -R m'R A - I A E T - ~ - S G C W l N I R A - g V R D E E F 8 l D E E F A W -
a n a c ll e ll +-lI
$ R s R E E I I I E I E s I s a s n E E E E 1 E a 5 m a 9 5 c Q c Q Q Q i i i E
S
@@ A c-i
.g
" =
E n E a y 2 5 EXl BX!
~I 'I lllC,' si ll m :i
- ll i .k);jj i
. @@ @@ 81 1
. &K! E l E ;! DXl O i! -
m l,i ij" l lj" mii 1 op! I li " " lg a oll= l
mann mee nam, . Iamt
"^" ".""*'" ma n m (i mna na [ , , , , , _
% x : do,%
% -++- I1j
, , , m- . , . ,
(. --- m. "a=axm anam um ,,,
., m anam rum,
,, f a TO m TO STH TO ]aD ,T NT% p M
" 6 ,
nunn i. l s 7-i i J3 \ -
- o \ / ~
varum , / a mma rune = ro uoena l l WO!"_/g x
*e m a n iimim
'bT.e m"as T
co maii
=n m"" ;L"', m,n=
o'sEs N m
~ <
O s 10 ISI,T MTR in wg _ , _ , , m anam rum, _,m k' . mnm anm ( i .- ca , M M d'.$
~
M am ucran M a,m,
-b4- -
gg, j.g a n "Ta
~~
me n m 4,, e .. ( .
... m _. ; TO 30 mouth TaNE Figure 4.4 Condensate System
( -) ( -
) C .. )
=l I - _- - __ __c 3- _ __ _ _ _ _ __ _ - c 3_ _ _ _ _ _ _ _ _ _y _ _ _
l J.J 8! ' E 5'$:
$! i limni ilS:ci E b)t " ---
h)t " - ft ft
& +- e-
- e{p ea
)
.H:
i 2 , _' if I
!- ' 4 i
- o. Q 5 I -@ $
3 ,!5 I i li : iid 0 l
. E e is ~ -
E e
! )[ ],'
r1
~
i; 2 ..
/ r4!,
s, i n 9 x
~
W )[
]
11 g i
=,
72 (( ll1
Pn O O m m l l ' g_ e" s ,m
,i T
NK EC ( VA 9 T
] OS T
p n l l l5l
'/
'3 ;C /E ,/ 3,
'c= l,=
'/
- ~ " ~ ; . .
4 f 4 _
. f A ,
R , s t E Rf F S r F F EE t t L L TG A N A A A AN H E - M H WA 0 M H T R 11 0 r a H T H T sC nX e t C U R U O uE i B I li l A A I I 0 s O N O S 13_ o A 0T CE H ' s o p m
- - _ - o
,1 C
- ; : s eT m e
1 I
" l} -
S t s y g a l I ' l J_ -
- i n
r l o
,,m
_ yT o e @ _ C r e t a g C."., 8 W
- S, - f N O
I T C O R Wg C
/T C
u l
/,
s ./," S t la P 6
@ Sy Cg W 4 s*
a- r
=
/T 1/T C
3 S R
-~
i F e r u g A ri-C 1
@, ,,m ;V
=.A
= E '
M d G 4 a o G Q 1 d a a d ' E R U-T
/T C Q d G T c 4 R1 U ~ '
T S Q Q d To 4 E-M A Wr T 1 l iI l L '
/T
' I .
il! i
'1 1h u((
I i a s t K g t A A 1 M jm a
~a a .s-,m -
Ili 3 Ill nis.
. . Ill .
l sp i lei r~~~m I I 3 O '
--C, , .. , n. ll ,
., ,, a l i e s L_ _ _J g I
E
$ 4' Z Z n
l l -l l 3 e ! I
. gl I I I ~
lli l lil 111 Ill
~5
( ( Zn Zs 5
- N I
lI! E8
. ,1 iI
, =
. +
=
g I h e{ . g o. O i t m , i e v Q $_ v D I I g , C x __
- g. .fi
#" m e
39 t s y S x l o . r g t n
=
w g_ C o e s
- e
#" i m
=
=
.O m u
n l a t n o o c V d n
= ,
- l a
a s c u i
=k a "
s e "' Yrc a h C m e 8 4 L Sr c c- e r u r e "@ - 9_ . . . i F g m g-al , q'
,i 1 d1 - " 2 a
e r
]e
- c n
a ' 2 b i I
=
n e W=. . e s I 2
> @ g=-
E s e f a
]
[' _e=- g _n o c a m
, eg4' ge ]
[' m u y, D= 3t . =mr c ] [' < g-
l 1 k J A a I QW W _ il ais in l1s,I-cB<. : BE: .e H= i l G W.
. as I a
~
Ea "E *! b __ g a> I " 5 I:- m 5 2 5:- in o E E I-2 E
'i e
E. .o- : j [I!
~
i Z 3 i Z I.
!I gnX II
( ( k. ism l' 1 l
si is si is a si ,,
. ~
l eH :In Zi M:la Zt N:la Zt "' l M:11 M ll H :II l T e- s . llI - - - I m :je:je:j
! ~
Ihie
@ @ @ 9 :9 ,,,,
i eg eg n , n I.! 8 e i i E E M ica si - M Eca h EI - Zi ri , < zi ri z z i e h Go - G- ; 33l2-! l i I a! Eik( N _EI eg EIb ., EI ! 5 l - , , _
.,. "@+ , t
). 7. ,
&i S:
ll1 s @'Y, . 0 x, hw ?
~ , -'
l I! i lj 1 L g( Q+) - + --- I lj[ u t
SANTIAGO 81 JL 220 KV NORTH EAST BUS , CONTINUED D Q TO SONGS 2 le 3 (M n
% n 1 4012 3 4022 5 4832 a -
H" : , I
> u u Hb Hb Hn
! 1912 4 8022 8 8632 Y Y cist0 w ici dsn ds swtien n
220 MV NORTH WEST BUS fg n
'iHilHH ' H il H H llGHTNING ARRESTOR MAIN TRANSFORMER MM WW AUX. TRANS "A" AUX TRANS. "8" AUX. TRAN1 "C" l NWW NWW NWW o WW WW WW I4AIN GUIstATOR If if If if TO TO TO T3 4110V 4160 V 4180 V 4tt0V 8USIA BUS 18 8USIC SUS 2C Figure 4.11220 KV System
1
~
L8 nn-8 7-) . ' gag 4TM"*":YesYds
- s ffg ,
*yI is'EaYdU j
ii lgi -
-*-7 ; uan ggg,=
- T + - O J a n*a Q u . u ' *a .- l 4 y
- I -4 i.4 u,
~
s 25 4, < h*'722 YS "gl5 ' g ++ h 4T?
, i y e-= ,
[ , m,e,n.- g 4$# 4 .-. ,
+yM -
E
-(-^ -
b-
-<eTW -
W WN Wi[A en
, 4p $
=
8pi . Il. ll ! ~4-T3
@Ja= 4 a _,J ~
I << [P
*b'A" 4T3 hudxEsanl
~
- h, g
} 5, a
.: > g
-4
! ->__ l 8!!
I udxm"EE h
. j* h' Cf E {
-+-TF
~
1= - e s gi
*p>- 'e li;.:
= + T-*--O dan *a l5 T &
@=82;g 4T3 =2i .
T4 1
=
=aac gE I B_ :
+ Q
+_ s -
sg g,g l g ,_.4_! ' g. R 5 i a n a== va 8 25 4 h, q s- =s f Q 9 l
+++ - >= .(y .
R5i--
~
-+-7 g_,_., ;- -*-i-+-e - >we=
- TQni."JM
!! 3
io acei m'" '" WTERMW est V SWGA .01 400 V SuuGI .0. 2 1119 1218 lb g 74 3ACC 2 _ - !' - I - TO O c - IAC$ tt - _ _ T0 0c asCC2B I I I I I T n I I I o nm) nise) nia) ntN) nin) nm) nm) nNi) I)nm) sAmnv mAmay sAmay amar
' $7*Cuaun cunun $7 57~ cemen CnAnun5 7 MTA SET S MT C MT0 ~~
m ufu . - -a __ __ __ __ __ __
" " =
vww ?.'"V wyw '!!'."V r- m m ., 9 - pd um. o i Es Er Es um v .a r I-s TO tsGNTIIBG
& Q TO MCC I
).c ).c ).c ).c ).c ') . ).c ).c 13 v AC DT.ITv BU5 ,IN VAC VITM OUS #4 _iN VJbC WTAL BUS M . IN V AC WTM BUS M IN V AC vvfM SUS E3 IN VAC vtTM BUS MA IN V AC VITM OUS 5,. IN V AC VfiM SUS 4 Figure 4.13 120 V AC and 125 V DC Systems
.E S N 0
t . aN TTO ESR L A m. GO T T= R
.S E
E gTmS R. a 0 rO aA m. GCN u s6O C G . I I n o it a L i z A LS n C l TC AR a l I FIO g I N C g ENS e HV I r E T SCDE A O T e i v t a r t T s NER S in EG DA NE R O N 4 0 i I l Sa OG I T Y A m EA RA I l TA I A g d AH R K PA EE TA SA R E P K A CI T O O e IVS it S 4 S 1 E C VSE 4 C ff A 0 N e N E 5 D r I I I GER u T N RA E eP ig I A tE A ER F R P g I I m . R Y Y r aft 0I T c 1R I U a T U P m A C
. E D a T E a
n SS li l :
T R R N ATEO EII T I LUNA CUIPR UPUE P N QOE n i o t a i z n a g r T O N * *
- E D L R O L* TL*
NOR f t TN O O R A O i FE RMS TOV I R T O T R T STN A h I H T I NOR I T A N R l I S-SN SO R I I ORE P O S E n A CP R E E C U CP P S O O O U S 5 1 4 e r u g i F R TR ATE N O T I ENMA LAPR C LI E UPU P N QE O s. o n. I ! I ,' ;
- s
. . ,s" ^
.-y , , . .. . .
' +
J r
- : _ .: g ,
\ . a
- . - - - *2 , .. ; .
v, . .' . , , y' . f
. &s ,
. , , . r t. 4
+ s L.- ,,
..ei.;,'.v'[,r-, -. r . ,. . ;" s .* 5.:
T - :. *
, e :.F gr.p. v; . , < Vg. - ,v m
_ _ _;g , . s. c y $s. .
, # es
. t w
v
. ' T %; _. -
s 4 pf ?,.h, .. ' , 4 w 'c . 4. , ' . s -; s. 1 .43e
' 9?q
.o . .
, , a
. ..: w, e / ,. ; L i0 - .' '-
5 3* ,; g . 1 J-gh., f . . ". .W p . . J.,. TN ,, ; ; ': l 7 ,- ; ' 1,,7. v' ..
- ) *#
^
,~ *
. g. Q _.;
1
- [_ " ; .. - l: , .,
A
?.-. , ;. ,. 1 s
':. i g., T. , ,, . . . *' < ..' ,. /* .,
4 yg.' .. g,; .j
, ,. 4 g1-.f , , >gF <
o
. ... . p . S.n. c .. -- ,
',
- q u
+
" fpN
,, , s J.rJ .. . ,
.<e. ! '.:f,: t. :
?
/
..f ~
.9 w
.'."'L, . ..-;g , , , ..i: . ..
*- ' 4, -*F +
x ?. s .v ' , . ' ':)
- s. ,
. %;,; > % ". n..t f ,. n- '. n;.C.+.
,5 ? e
?.y-1
.f.. i,.,.,. .se .
. . . #g . 4 +.. ..
; ~a.
.f - : .5.,.,f .
. c:p : , p' ,; l- w _,p .
j . . r. % Q j [.g ,i '
'%, 5 .: ,
{I.h:., .
-. ,: [ ' ^
.'c ,. ?y ,, , '- + m g ~-
} . ,. 37 _ sp -:c ,
.., ;, , ,'f.va-l :~
- i. e..4 3.. ;;? . . . , ' ' ' _ v. . -; ,,,,
.,.,,0 g c U, + %* . .
..1 J j;,.',;. .
eh wD _ o e '. - .
'y m r . . . ;, -
y
.- ,.. '4g .;,g]. '.g v5
^
; .. y }'_
/.$ , .% g .. , *'
.,i3.t.33 cg g.3,g ,. 3; ,.- 1"3
+ ,r 7,., e . _ : ; c.g
* + D ,e . .<
-9; , E. t49,( ~ . ;.. y f. 4^
, a, s.f
-- /.; s.y.
- 4. . ; . g,,
f,yc g s . t-
.,.c T. ; ,,-
.. 4, ,; ,y.c
.. v .g $g > !}: 3~. .M
;.. ,. y? ..cr . .p y .;
3,. r'(g. y)g y.;. ,. ,C. g, , , i .
, . , , ,y . .,,,
" . . ll p q, r "q, s. {;\.J
' i' ,Q s
.: .. .a , , . . , ,,1 u U ,.>.e.1
. , , . ,. /.r.,,. e, ,( ,w g......- .f ,,' ,
%. ;,; r , .g , . p . % '.0, , s.p. .sp g lM,y - . . .
r:{ v u..,s M
. < ,4
, . ., . x r. .
. [.t . . w ,4.:.; .e. ~-
n :;j' - ,.
;, .s . x s.,my + C.
- t. ;
(f. 'r 3
,. ' m.1.,
.',i p ,.v', ,ke- # ; . '
+
. ,1:.~',.
7
,.s% .. ,.:.,,,..,
,, 7 > :?g . , . , 4 <. . ,:. ~ . .. ,. ;
r.4 . .
' g .
. . ..s : ,
-l* g.A"jV 6,j +.. w,,,
.,"..g 3 ' .[, " .
3
. ., . , t. ;- 'p
; .]d 'y g j @y p. .h J.,g' . g p,..g g y) l l.'(
. q,f, ' ' j. l
, ^ l :. le U U ~" u Nif. Q 3: & mY.w. f, m(c.
/
.t. ,
i cp & - ~: . d uCJ,Q/msyenU e . E O w'% &;f.GE.;blB Q. 4' ; , x M *y., o;y, m o m.,, v,.. m y'.. y'
=
.w.
' ^ k,j;...?,,g y ':, .~..
; _Q1 . ? % 3 ;'. *e sf.? . ,.
- J ' N W ** * * - .Y'* *b,h h La M i. i$ o g>~r.-{N' T} < fT 15
' 4.l,M.,- f * ' : ' # , wg o
; - , a.: .1,p.1 ' Pxn '
. ,* g' o Y Y
.N:f ;" '..) T
_ }4 i Qi '4.p.ff'.'&y.h*.:..y$:a h' +?% )C r . ,/ ' n" ^f ' .
- . n.. nc :. x s, . ' r r. .
i: - Q&
*?
. ? =
A t??'@Y t *$., o
,9 .
.- i 7 ; . , .
*, $,h.. '. ' . . . , 6 ya .# 'yY h b,'.;;,%:i{ .'?
- ;. : +*
> Q . *-
..r .
.iw* yR,.s.g,.x. -
Y - p .; 4 :.* i- M. ,.y.p['.4 q%:.\, % Y. ',(o (.5Q y;. . , , . s- 9. c.,' ' 4;4
;r s - , .- .*
2 . . , C 49.'n ;.< +
- ve .
L. ,*p..? " c - y): ( ~ g % .'
=
3.s
- .f
,_ Cf. s . '. ' r . .'k; -
,q ;'4' Q
.? '-
. $r-p -
.,_ :,,) j
.,i; p . t %ll.' 3 .,
- e.
g,.:. e :. g 3q. ...p., .s p , 'Mp jyr.j^L}. y / 9
. U3 0
-P .w . , . . . ..
Ifc. . . ,..
'n
-- . . :. .. . a '4 p
- [. .lyj. .q I .r ,.y ,.,, 5 %..
.,f y{,3 3 .;g- 4 ,j .
. v>
s q,, Q 0 . , ' r. T s ? ,3 j. . . /.y
~
, . , . a.: . , . . .
. _ y.% w 06 .~-
i ...'. .g . m,
\.
-q.
. . . .- ). [. .p g, ..,.-
..). . 4..,
-: .;:. s p
, . ;lg;+_,
,q .
's,.
. 9 . ? ", - (q. 4;. ,
3.:., e" :. 4 '
,- . .- f. <v- .-
- . + .
.., sv- g.;- .}w . .
. m s
g.
^ -
s .. + . o, ., y ;.: .
.c,
; ,; . 4,1 .+ #,n 4
, , ' . A- .
x . u,, g . . . . ', *y,,p. :1..;. ,.q s ' %q- - a y
, A, L _ ' *
* -, -i, .g , V
.~r , l .[ . N t. , N/f
,g r,, .t ..
f 3
- ' ' l, ' ..
?( ,.
[
- . v. ..a.-
. , m, , ,. ...
. y , ,
}e l
'-f .p
4 a u ?v
. ,. .y p.
., ..__ y '
w.
+
* \ .,k%
',w A .
p .- .lq . .%:
.. ' '.~' 'wt. Y0@h q ll*{' ' ' - 4 '
, f- .' > ;'N,7
,g z g h.'" # , J ~' g ?]' -l . * ,
*h
,m i t IT ' 7 3
t, N O I T. h _ A. sy p p a lm. m. i g g u
" a n O C
E
- . r
. t ; , E ;.
- r D I .
,EE
,T c, A .
,W s
,T
,O"m I
I' mE M s I Y n e I u s ic a u n s f f r r O
/
a e a r r m . t u u n s n m e t a c a '" . c a n u
"" j
=
c s d n e, , n A eg r u s d o, n a m 1 r> f o o yIi Em Ia c R lo
- y c
c_Rr, En D. o 3a r .. t r n
- ;w o
.a a
m "n s o r.. .
. C e m N s
- o . 7 l
m s a
.TN 1 SsE ITEH ,m o s .OC
. 4
, nu n .
mMO ,a u i e
.N T
T ,i s o r Mse E tsA O 6, o v i f a r . u rs0 I T
,,u
,r s su v .
OA. _A L ,c f a r e o .T m ig C r n F
, , _I /ll I a wu T s a
i s iC W D
. e, T n n u
.w
*l* - k u
* * .T
.M s
u u m e ;* L ;a o , _D E C
, / c s t
n e8.* f
.E /l s a p
/ ",o u
a u e .R
. n c n
I/1 i m ( /- e s L E A A r
.F
.An
.s E \I i '
C 8,r u s
" ,,,sur /
,m u
c a M c Tu E C _Aa s E M,. a T e A S e i . .. .c
.. m n
_D- D W
- }'
m e _e
.s.
n n o i
. ,m
- u D X (l
. TC
..E I
sA _ -
-.7.".
* . 1 l
=l' o
- si
5 LOSS OF POWER EVALUATION 5.1 Electrical System Operations Leading To Auxiliary Transformer C Differential Relay Trip On Wednesday, November 20, 1985, at 23:50, an alarm indicating a ground on 4160-V bus 1C was received in the control room at SONGS Unit 1. The ground detector voltmeter indicated a 100 percent or full scale ground. In response to the alarm, the operators referred to Operating Instruction (01) S01-9-7,
- which deals with 4160-V and 480-V bus and feeder faults. The procedure first has operators attempt to locate and isolate the ground fault in the loads con-nected to the bus. This required operators to shift services to redundant equipment powered from sources other than bus IC, and stop equipment powered from bus 1C. For example, the north charging pump connected to bus 2C was started and the south charging pump on bus IC was stopped; the operators veri-fled that the ground detector indication did not clear, and then restored the original equipment alignment by restarting the south charging pump and securing the north charging pump. In the case of service station transformers SST 1 and SST 3, alternate supply paths were established to energize the respective 480-V buses prior to disconnecting the SSTs. (Refer to Figures 4.11, 4.12, and 4.13 for details of the electrical distribution system.)
By 00:20, November 21, all load circuits on bus IC, except for that of the north circulating water pump and the west feedwater pump, had been tested; however, the ground fault had not been located. Unit I was operating at 250 MWe at this time and the operators received permission to reduce load to 150 MWe in order to stop the west feedwater pump. One of the steps required by the procedure (501-9-7, step 6.2.7) is to check the fuses of the ground detector potential transformer (PT), because a blown fuse will cause a spurious ground fault indication. Itsis step was to be performed if the ground on the bus could not be located by deenergizing all feeder circuits. In this instance, the operators proceeded to take this step prior to isolating the two remaining loads. The PT fuses were verified to be good and the PT was returned to service. At 00:24 the unit was down to the desired 150 MWe, Around this time, the con- . trol room operators closed the bus 1A-1C tie breaker 11C01, thus parallelling auxiliary transformers A and C in order to verify that the ground fault was valid and not just a problem with the bus IC ground detector. With the buses , parallelled, the ground detector associated with auxiliary transformer A also indicated the fault, thus confirming the existence of the fault. The buses were para 11elled for approximately 5 seconds this time. At 01:30 the south circulating water pump on bus 2C was started (it had been secured earlier to enable a search for salt water leakage into the condenser), and the north circulating pump on bus 1C was stopped. The ground did not clear, and the north pump was restarted. At 02:40 the south circulating pump was again stopped due to increasing chlorides in the steam generators. At approximately 02:45, an electrical technician was given approval to check the ground detector on bus IC. The technician measured the voltage across the 5-1
ground detector relay and confirmed the voltage to be 215 volts (210 volts is nominally 100 percent on the ground detector voltmeter), thus verifying that a ground fault existed and that the ground detector was functioning properly. The technician informed the Shift Superintendent of his finding. Following discussions with control room personnel, the technician checked the PT associated with the ground detector for ground. To do this, the tie breaker between bus 1C and bus IA (11001) was again closed. At this point auxiliary transformer C and auxiliary transformer A were again operating in parallel, connected to the inter-tied buses 1C and 1A. The ground detectors of bus 1C and of auxiliary transformer A both indicated the ground fault. The PT of bus IC was then disconnected, but the ground was still *
- indicated on the other detector. This test eliminated the PT as a possible location of the fault. The PT was returned to service. At this time, 03
- 35 the auxiliary transformer C source breaker to bus IC, 11C02, was opened by the -
control room operator; bus IC remained energized from bus IA, and the ground indication on both indicators cleared. (In this instance, the two transformers remained in' parallel for approximately 2 minutes. ) This action isolated the ground from bus 1C and indicated that it was somewhere between the X winding of auxiliary transformer C and its circuit breaker 11C02. After further discus-sions between control room personnel and the technician, the technician checked the synchronizing PT of bus 1C to further isolate the location of the fault. To do this, the source breaker 11C02 was reclosed, again paralleling auxiliary transformers A and C for approximately 5 minutes, and the synchronizing PT was disconnected. The fault did not clear. The PT was returned to service and the source breaker was re opened from the control room isolating the fault. Having determined that the fault was not on bus IC and that the west feedwater pump would not have to be deenergized, at 04:00 the control room operators commenced increasing power and by 04:30 the unit was again at 250 MWe. Meanwhile, the i electrical technician and crew were visually inspecting the electrical circuit between auxiliary transformer C and bus IC, the area around the transformer and the X winding's reactor and associated bypass breaker. They were preparing to rack out (remove) the reactor bypass electrical breaker after receiving permis-sion from the control room. At this point, the electrical distribution system status was as follows: Auxiliary transformer C was energized and supplying 4160-V bus 2C; i.e., circuit breakers 4032, 6032 and 12C02 were closed, i Generator at 250 MWe and breakers 4012 and 6012 were closed,
; Auxiliary transformer A was supplying 4160-V buses 1A and IC; i.e. ,
l breakers 11A04 and 11001 were closed, - i Auxiliary transformer 8 was supplying 4160-V bus 18, SST 3 and associated 480-V bus 3 were being supplied by 4160-V bus 2C through breaker 12C11, - Lighting transformer was on normal supply off bus 10, South circulating water pump had stopped and both feedwater pumps
.! were running.
120-V ac utility bus was on alternate supply off the lighting switchboard, which in turn was being supplied by 4160-V bus IC, and All other electrical system alignment was normal. i 5-2 _ _ _ - - _ _ _ _ _ . _ . _ - _ _ _ _ _ _ _ _ _ _ . _ _ . _____ _ _ _~ _ . . _ _ _ _ _ ._
5.1.1 Relay Trip At 04:50 auxiliary transformer C tripped on actuation of transformer differen-tial relay. Later observation at the relay panel confirmed that the differen-tial trip involved phase 8 and phase C. Phases 8 and C trip targets had dropped, indicating that a fault current in excess of approximately 1500 amperes was involved. Control room operators received multiple alarms when auxiliary transformer C tripped, among them the turbine first-out alarm for auxiliary transformer C differential trip. Upon loss of the transformer, 4160-V bus 2C and all loads . connected to it, including vital bus 4, were lost. Diesel generator 2 automati-cally started upon loss of voltage to bus 2C. The control room personnel observed the loss of vital bus 4 by the loss of its potential indicating light with the , green dot on the vertical panel. Based upon this, the reactor was manually scrammed; the unit trip pushbutton was also manually actuated. 5.1.2 Discussion In responding to the ground indication on 4160-V bus IC, the operators correctly followed Operating Instruction 501-9-7 in attempting to locate the ground on the load circuits of all connected loads, up to the point which required them to de-energize the feedwater pump motor circuit. The 01 did not instruct the operators to use the ground detector on an unfaulted bus (bus 1A ground detector, in this case) to verify the authenticity of the detector on another bus which is indicating a fault. Nor did the 01 direct operators to tie a faulty bus to a functioning bus as a means of transfering power or load. Yet the operators did close tie breaker 11001 three times when a ground fault was present on bus 10. When bus 1C and 1A were thus connected, auxiliary transformer A and auxiliary transformer C (X winding) were operating in parallel. The transformers were operated in parallel three times for periods ranging from 5 seconds to 5 minutes. Operating two transformers in parallel supplying a faulty 4160-V bus is not appropriate for the following reasons:
- 1. The ground fault on the ungrounded delta-connected transformer secondary is being supplied by a neutral grounded wye-connected transformer secondary, thus providing a fault current path back to a source.
- 2. The 4160-V switchgear is not rated to handle phase-to phase or short-circuit faults should they occur while the buses are energized
, from the switchyard through two transformers in parallel. (This is the same reason why current-limiting reactors are installed in auxiliary transformer C leads while the diesel generators are run in parallel with the transformer.)
- 3. Feeding a fault from two sources will exacerbate the situation and could lead to the fault tripping both sources.
- 4. Parallel operation of transformers, especially ones with unequal Impedances, would lead to circulating currents being set up between them, 5-3
l l 01 S01-9-7 did not include specific warnings regarding parallel operation of transformers and other sources while a fault exists in the electrical distribu-tion system. (Momentary parallel operation between functioning sources should be permitted; however, where continued parallel operation is required, the system should be designed for it.) 5.2 Unit Trip and Loss of All AC power On depressing the unit trip pushbutton, the following events occur. the turbine trips the generator exciter field breaker opens ' the 4160-V bus 1A source breaker 11A04 opens
+
4160-V bus 18 source breaker 11804 opens an auxiliary relay (UT-X) energizes which in turn trips generator - output 220-KV breakers 4012 and 6012 (and energizes another auxiliary relay for local breaker failure backup protection). During the event, the unit trip pushbutton was depressed 19 seconds after auxiliary transformer C tripped. At this point, when breakers 4012 and 6012 opened and power from the switchyard through auxiliary transformers A and 8 to 4160-V buses 1A, 18 and IC was isolated, all ac power to the in plant ac distri-bution system was lost. Only the plant de system and the inverter powered vital buses (all battery powered) remained operational. The utility bus was now without power, The 4160-V bus 1C undervoltage relays actuated and, with the existing loss of bus 2C, initiated generator lock-up and the loss of voltage auto transfer scheme. Both emergency diesel generators received a start signal and started (diesel generator 2 was already running from a previous signal). The control room operators were following Emergency Operating Instruction (E01) S01-1.0-10 for reactor trip or safety injection. Step 3 of this instruction requires verification that buses 1C and 2C are energized. With these buses deenergized, the operators turned their attention to the loss of all ac power instruction, 501-1.0-60. Step 1 of this E01 requires verification of offsite power at the 220-KV switchyard, which the operators did by reviewing the vol-tage indication in the control room. Step 2 requirer a check for SI initiation on the reactor panel first-out annuciator window 2 (RPF0-2). The alarm window was lit, indicating SI actuation; however, this was a spurious alarm and the operators were aware that it would come in on loss of ac power. (See section 8.2 for an explanation of this spurious actuation.) Operators in the control room then checked the SLSS load group lights on the SLSS surveillance panels located on the vertical diesel generator boards and observed that they were not lit, - indicating SI had initiated. (See section 8.3 for a discussion on the spurious indication of SLSS load group lights.) The operators then checked the status of pressurizer pressure and containment building pressure, determined that an . SI was not warranted and concluded that $1 had not initiated. At step 3a of the E01, the operators are expected to wait for the automatic operation of the loss of voltage auto transfer scheme to complete and the end-of-sequence light to light. However, the end-of-sequence light did not illuminate. An operator depressed the unit trip reset pushbutton, to enable closure of 220-KV circuit breakers 4012 and 6012, but the reset did not occur because this action was apparently taken before the generator 18-KV motor-operated ) disconnect (MOD) opened. The operator at the electrical board checked the i status of the generator MOD, which was, by then, open as required, and the ) i 5-4
status of the source breakers to buses IA and IB (11A04 and 11804) and the tie breakers of buses 1A-1C and 18-2C (11C01 and 12C01), which, contrary to expectations, were not all closed. The E0I did not address this particular situation and the control room operators decided to recover power from the switchyard to feed the 4160-V buses one at a time. They attempted to close 220-KV circuit breaker 4012. The first attempt failed because the operator did not depress the synch check bypass pushbutton (see section 4.12.2.1 for a des-cription of closing operation of this breaker). The second and third attempts are believed to have failed because the unit trip reset had not reset the gener-ator lock-up, and treaker 4012 tripped free when the two attempts were made. (These two attempted closures and trips of breaker 4012 are seen in the oscillograph traces.) Af ter the failures in closing breaker 4012, the operator at the J-console again pushed the unit reset button. The operator at the vertical panel tried to close breaker 6012. The first attempt failed because he , did not depress the synch check bypass button. The second attempt was success-ful, following the activation of the synch bypass button. The operator then closed breaker 11A04 energizing buses 1A and IC (bus tie breaker 11001 was already closed), followed by closing 11804 and 12C01. Thus, four minutes after loss of all ac power, power was returned to the station distribution system. The evaluation of the operator response to the unit trip, loss of all inplant ac power and spurious indication of safety injection actuation, is discussed in Section 7. The operators were expecting the end-of-sequence light indicating the comple-tion of the auto transfer scheme 2 minutes after loss of ac power. However, based on a review of the scheme, the time required could be closer to 4 minutes. The actual time required will have to be determined during the ongoing trouble-shooting of the transfer scheme. The operators' lack of knowledge regarding the timing of this scheme points out inadequacies in SCE's previous test program and in the training operators received on the response characteristics of this system. The difficulties experienced by the operators during their attempts to recover power from the 220-KV switchyard indicate inadequacies in their training and understanding of the following:
- 1. Unit trip and generator lock-up reset.
- 2. 220-KV circuit breaker closing operation, with and without synchronizing check.
- 3. Loss of voltage auto transfer scheme.
Although both diesel generators were ready and available to supply emergency power to the station during this period, the operators did not use them since 220-KV offsite power was available. Had the operators been unsuccessful in closing either circuit breaker 4012 or 6012, they stated that they would have used the diesel generators to supply power to the station. The procedures are not clear as to when this is to be done. Just prior to auxiliary transformer C differential trip, the electrical techni-clan was given permission to rack out (remove) the transformer X winding reac-tor bypass breaker. Had he done so, the control room operators would not have been able to close the output breaker of diesel generator 1 because of the elec-trical interlock between the two breakers. A review of the interlock circuit shows this to be a design deficiency. 5-5
6 WATER HAMMER EVALUATION This section discusses the water hammer which occurred at SONGS-1 on November 21, 1985, its underlying causes and the damage incurred. Since failed check valves in the feedwater piping were the underlying cause, this section also discusses valve maintenance and inservice testing related to these valves. To clarify
. the discussions that follow, a brief review of water hammer phenomena and com- '
monly accepted definitions are provided.
, Hydraulic instabilities occur frequently in piping networks as a result of changes in fluid velocity or pressure. Some of the better understood occur- j rences include induced flow transients due to starting and stopping pumps, opening and closing valves, water filling voided (empty) lines, and pressure changes due to pipe breaks or ruptures. As a consequence of the change in fluid velocity or pressure, pressure waves are created which propagate through-out the fluid within the piping network and produce audible noise, line vibra-tions and, if sufficient energy transfer occurs between the pressure wave and the pressure boundary, structural damage to piping, piping supports and attached equipment. More specifically, this pressure transient is a fluid shock wave in which the pressure change is the result of the conversion of kinetic energy into pressure waves (compression waves) or the conversion of pressure into kinetic energy (rarefaction waves). Regardless of the underlying causes, this phenomena is generally referred to as water hammer. The more severe the out-come, the higher the likelihood is of the event being called a water hammer.
Water Hammer Definitions
- 1. " Classical water hammer" generally identifies a fluid shock, accompanied by noise, which results from a sudden, near instantaneous, stoppage of a moving fluid column. Unexpected valve closures, back flow against a check valve, and pump startup into voided lines where valves are closed downstream are common examples of underlying causes leading to " classical" water hammer and are generally well understood.
Analytical methods have been developed to predict loads for this type of l fluid hammer and include the effects of initial pressure, fluid inertia, l . piping dimensions and layout, pipewall elasticity, fluid bulk modulus, valve operating chiracteristics (time to open or close), etc.
. 2. " Condensation-induced water hammer" results when cold water (such as auxiliary feedwater) comes in contact with steam. Conditions conducive to this type of water hammer are: an abundant steam source and a long empty horizontal pipe run being refilled slowly with cold water. The cold water will draw energy from the steam with the rate of energy transfer being governed by local flow conditions. As the steam condenses, addi-tional steam will flow countercurrent to the cold water, and as the pipe fills up (i.e., the void decreases) the steam velocity will increase, setting up waves on the surface of the water, eventually entraining water and causing slug flow. Slug flow will entrap steam pockets and promote 6-1
significant heat transfer between the steam and colder water. Figure 6.1 illustrates (in simplified form) the flow conditions which would come about during the refilling of a voided horizontal feedwater line. Once slug flow conditions commence, a steam pocket will suddenly condense, creating a localized depressurization instantaneously. The resulting pressure imbalance across the slug (approximately 700 psi at SONGS-1) causes the i
' slug to accelerate away from the source of pressure and toward the region of condensation.
Condensation is extremely rapid and predicting the exact location is im-possible. When the water slug suddenly strikes water in a previously filled pipe, it produces a traveling pressure wave which imposes loads of
- the magnitude that would be induced by " Classical" water hammer in the i piping network. This phenommon, called " condensation-induced" water hammer, occurred at SONGS-J. -
i Predicting loads associated with this type of water hammer is extremely difficult because of the interactive and complex hydrodynamic and heat 1 transfer phenomena which proceed the sudden condensation. Void fraction i (or how empty the pipe is) and subcooling (or how much colder the water is
- than the saturation temperature of the steam when steam and water come in j contact) are two important parameters currently used in models for pre-dicting this type of water hammer occurrence and its associated loads.
- 3. " Steam generator water hammer" (SGWH) is a condensation-induced water hammer which has occurred principally in pressurized water reactors (PWRs) with steam generators having a top feedring for feedwater injection. The underlying causes are similar to those discussed above (e.g., voiding of the horizontal feedring and feedwater piping immediately adjacent to the steam generator and subsequent injection of cold water). Damage from SGWH has generally been confined to the feedring and its supports and to the steam generator feedwater nozzle region. However, damage to feedwater line snubbers and supports has also occurred at other plants. A SGWH resulted in a fractured weld in a feedwater line at Indian Point Nuclear l Power Plant, Unit 2 in 1972.
1 1 6.1 Plant Conditions Leading to Water Hammer 1 i Plant conditions at SONGS-1 which led to a steam condensation-induced water hammer included the voiding of long horizontal lengths of feedwater lines
- (which allowed for a backflow of steam from all steam generators before opera-
, tors isolated the FW lines by closing MOVs-20, 21, and 22) and the subsequent
- 2 reffiling of the FW lines with relatively cold (i.e., less than 100'F) AFW.
i Table 3.1 highlights approximate times and events which are pertinent to esta-blishing water hammer conditions and which are referred to later in this dis- + cussion. Figures 4.3, 4.4, 4.5, 6.2, 6.3, and 6.4 describe the flow circuits, , valves and other equipment affected by this water hammer. l i Upon detection of the fault on C auxiliary transformer, relay protection de-onergized 4KV bus 2C, de-energizing the east-side main feedwater (MFW) pump FWS-G-3A (Table 3.1). The continued operation of west-side MFW pump FWS-G-3A, due to the unusual electrical alignment, plus the failure of the east-side MFW pump discharge check valve FWS-438 to seat, resulted in the overpressurization and failure of the east flash evaporator tube and shell side. The subsequent 6-2 l
1 [ unit trip de-energized the west-side MFW pump and denied power to the electric-l driven auxiliary feedwater (AFW) pump AFW-G-10S, With the cessation of flow to l the steam generators, the failure of check valves FWS-438 and FWS-439, and the l failure of the three check valves in all the feedwater control circuits (valves l FWS-346, FWS-345, and FWS-338), a path was provided to blowdown all three steam i generators through their respective feedwater lines to the atmosphere through the failed flash evaporator. The drop in the steam generator water level following the unit trip initiated the AFW system, but the electric pump was de-energized and the steam-driven AFW pump AFW-G-10 took 31s minutes to deliver flow, because of a programmed warmup period for the turbine. Thus, for 3 to 4 rufnutes no flow was being provided to the steam generators and the leaking check valves permitted the horizontal feedwater lines to void. Further, the initiation of AFW flow at about 135 gpm
' from the steam-driven pump was not effective in halting the voiding, because flow was being carried away from the steam generator by the steam blowing down from the steam generators, through the failed check valves in all three FW con-trol stations and out the leak in the flash evaporator.
Figure 6.5a presents data from temperature recorder TR-456, which records the temperature of fluid in the feedwater line in the common header just upstream of the FW flow control stations. The recorder and pen failed due to the loss of station power but, following re-energization, indicated a sharp rise in temperature (indicative of the presence of steam in the FW line), followed by a rapid drop in temperature (indicative of the injection and mixing of AFW in the fluid passing the sensor), followed by a tailing off of temperature (indicative of the closure of the motor-operated isolation valves). In addition, tempera-ture recorder TR-96 (Figure 6.5b) data from sensors upstream of the east and west MFW pump suction locations demonstrated a similar temperature increase to about 385 *F and then a drop in temperature. In order to have the rise shown by TR-96 (which is about 20 feet lower than TR-456 and has an intervening volume of about 2500 gallons of 300* to 365 *F water) a blowdown of a significant portion of the feedwater lines upstream of the FW regulating valves must have occurred. Following restoration of unit power, the motor-driven AFW pump (AFW G-105) started automatically, increasing the indicated AFW flow rate to a preset rate of 155 gpm per steam generator. However, all three steam generator levels continued to drop since the FW check valves remained open, the main steam sys-tem had not been isolated, and steam generator blowdown had not been isolated. Subsequently, operators, following an emergency operating procedure for reactor
, trip response, isolated the failed FW check valves by shutting the three FW control isolation valves, MOV-20, 21, and 22 at approximately 04:55. Isolation of the feedwater trains occurred before the water hammer in loop B.
( Subsequent to the isolation of the main FW loops, and recognition in the control room that both AFW pumps were delivering water, the operators become concerned for the over-cooling of the primary reactor coolant system and the decrease in pressurizer (PZR) level, at which time operators decreased AFW flow from 155 gpm to zero, and then back up to 40 gpm. The estimated time for AFW flow reduction is 05:00. Thus, refilling the FW lines downstream of the flow control stations was halted and then resumed at a much lower flow rate. This decrease in AFW flow rate adversely influenced the pipe refilling characteristics, particularly in Loop B. 6-3
The slow refilling of the FW lines within the containment building continued from approximately 05:00, when AFW flow was first throttled back to when the water hammer was reported to have occurred at 05:07 by a plant equipment operator. As noted previously, conditions conducive to steam-condensation water hammer in the feedwater lines were present for quite some time. The gross failure of upstream check valves which permitted water to drain from the feedwater lines and be replaced with steam was the underlying cause for water hammer. Leaky check valves have been previously cited in reports of other water hammer occurrences. Five check valves are known to have been failed at SONGS-1 on November 21. Estimates of feedwater line voiding and refilling are discussed in Appendix 8.
- Figure 6.6 illustrates hypothesized refilling conditions in loop 8 prior to the water hammer. The much larger voided volume of loop 8 and manual control of AFW injection following closure of MOVs-20 -21, and -22 (particularly total -
stoppage of AFW) were major factors in establishing conditions which triggered this water hammer. Appendix 8 discusses the refill aspects in more detail and also provides estimates of pipe void conditions when water hammer occurred and provides estimates of water hammer loads based on damage done to piping supports. Post-event water hammer load calculations estimate that a 1-2 percent void existed when the steam pocket collapsed; the prnbable location of that steam pocket is that shown in Figure 6.6. 6.2 Water Hammer-Induced Damage i The next four sections detail water hammer-induced damage to Loop B feedwater piping and supports, the FW Loop 8 flow control station, and to Loop 8 auxiliary feedwater (AFW) piping and feedwater system check valves. 6.2.1 Piping and Piping Support Damage Damage to the Loop 8 FW piping was confined to plastic yielding of the NE elbow and to a visible crack on the outside of the pipe, extending approximately 80 inches axially. The crack penetrates approximately 30 percent of the pipe wall at its deepest point from the outside and approximately 25 percent on average. Damage to supports, in some instances, was severe. This section provides a narrative and pictorial description of the damage visible after the FW piping insulation was removed. Figure 6.7 shows the FW Loop 8 piping layout and identifies the piping support stations where damage occurred. This figare also provides directional orienta-tion and indicates piping dimensions. Figure 6.8 shows principal areas of
- damage, indicates how the pipe moved, and highlights the narrative and picto-rial "walkdown" which follows. Table 6.1 provides a narrative description of the piping and support station damage shown in Figures 6.9 through 6.30, and -
identifies those support stations. The water hammer forces were sufficiently large to damage pipe supports and piping and to transmit loads through the containment building penetration structure outward to the B Loop feedwater regulating station. No damage was evident to the steam generator 8 feedring or nozzle region that can be attri-buted to water hammer, nor was there evident damage or movement to the piping between support H00C and the steam generator 8 feedwater nozzle (Figure 6.9). 6-4
6.2.2 Feedwater Loop 8 Flow Control Station Damage The water hammer originating in the feedwater line within the containment build-ing generated a water slug which transmitted a pressure wave upstream to the Loop 8 flow control station. Check valves FWS-346 and FWS-378, downstream of the control valves, were designed to prevent backflow, although post-event in-spection revealed that the closure disk for FWS-346 was laying in the bottom of the valve chamber. Thus, any closed valve would be subjected to the water hammer loads. In addition to check valve FWS-378, the flow control valve (FCV) FCV-457 and motor-operated valve MOV-20 were subjected to the water hammer loads, because they had been closed by operators following procedures in the Emergency
. Operating Instructions (E01).
Figure 6.30 shows steam generator 8 feedwater piping and valving at the Loop B
' control station (outside the containment building). Figures 6.31 through 6.34 illustrate the piping and valve conditions found after the occurrence of water hammer. The 2-inch bleed bypass line suffered minor damage. Figure 6.35 illustrates the relative positions of the main FW (10-inch line) and FW bypass (4-inch line) flow lines and the damaged check valve locations.
Because check valve FWS-378 was intact and operational, it was subjected to water hammer loads and absorbed much of the water hammer energy whereupon the bonnet studs yleided and the gasket was forced outward against the studs (Fig-ures 6.36 and 6.37). The failure of the gasket relieved much of the internal pressure, thereby minimizing damage to other equipment and valves at this sta-tion. However, FCV-457 did incur damage to the flow actuator yoke, as shown in Figures 6.38 and 6.39; and disassembly revealed a bent valve stem. Further discussions and illustrations of valve conditions (which were found upon disassembly following this event) are provided in Section 6.2.4. 6.2.3 AFW Piping Damage The AFW injection point to the main feedwater piping at SONGS-1 occurs in the
" breezeway" upstream of the containment building steel shell, as shown in Figures 6.40 and 6.41. The AFW lines run horizontally and then vertically to tie into the main feedwater (MFW) line. Figure 6.40 also shows the Loop 8 containment building penetration junction.
Water hammer loads were imposed on AFW Loop B piping, as is evident from the pipe motion recorded in Figures 6.42 through 6.45. Although pipe movement extended several hundred feet upstream, there was no evidence of piping damage. l 6.2.4 Valve Malfunctions and Damage Post event disassembly and examination of valves that contributed to water 1 I hammer conditions confirmed that check valve failures were the underlying causes(s) for the occurrence of water hammer. Inspection findings identified the following valve conditions: Valve Description As-Found FWS-345 MFW Reg Check Disc separated from hinge SG A arm, disc stud broken (threaded portion). 6-5
I j Valve Description As-Found i FWS-346 MFW Reg Check Disc separated from hinge l t SG B arm, disc stud deformed. l FWS-398 MFW Reg Check Disc nut loose. Disc SG C partially open. Disc caught inside of seat ring. FWS-438 FWP Discharge Check Disc nut loose. Disc partially open. Disc . ! caught on inside of seat l ring. FWS-439 FWP Discharge Check Disc nut loose. Disc I partially open. Anti-rotation lug lodged under hinge arm. The following observations were common to the MFW regulator check valves:
- 1. Severe wear was evident under the hinge arm; the wear spots were bright. The wear was caused by the disc anti-rotation lugs rotating against the hinge arm.
- 2. The surface around the hinge arm stem hole contacting the disc nut was recessed, as if the nut had been ground into the hinge arm forming " ball and socket" like surfaces.
- 3. The underneath surface of the bonnet stop was heavily hammered.
- 4. There is evidence that the disc nut was pinned to the disc.
- 5. All the valves had anti-rotation lugs welded to the disc.
The following observations were common to the main feedpump discharge check . valves (FWS-438 and FWS-439):
- 1. The disc nuts were loose.
- 2. The discs were partially open.
- 3. The disc nuts and washers were in place.
- 4. Neither the disc nut nor the disc stud was drilled to accommodate the locking pin.
- 5. There is no evidence on the disc stud of its being hammered against the bonnet stop. .
The following sections illustrate valve damage. 6.2.4.1 Swing Check Valve FWS-346 This 10-inch swing check valve in feedwater line 8 is the first backflow barrier between SG B and the Loop 8 flow control station. Figure 6.46 illustrates a typical flange end swing check valve design of the type extensively used in the SONGS-1 feedwater system. Disassembly of FWS-346 revealed that the closure disk was laying in the bottom of the cavity, as shown in Figure 6.47. Thus, there was no back flow barrier 66
t to prevent drainback until the operators manually closed valves MOV-20 and FCV-457 at approximately 9 minutes into the transient, i Figures 6.48 through 6.54 show FWS-346 components and the valve body cavity. l The seat surface of the disc face did not exhibit noticeable damage (see Fig-j ure 6.49), although there were two cuts (on the mating face in the valve body) as seen in Figure 6.48. Damage to the disc nut pin is shown in Figure 6.50 and the worn hinge pin hole is shown in Figure 6.51. SCE personnel stated that the nut had previously been pinned and evidently worked loose over a period of time. The worn pin hole and damaged disc stud are evidence of continual opera-tion with a loosened or lost nut. Figure 6.52 and 6.53 show the bottom side of
, the bonnet and the disc stop cast into the bonnet. Damage cauwd by continual impact during operation is evident. Figure 6.54 is a composite photograph of the FWS-346 valve cavity, looking towards SG B. Scratch marks at the bottom
- support the hypothesis that the disc had been laying free in the bottom of the
- valve body for some time.
L , As noted above, post-event inspections of the check valves in feedwater loops A j and C (FWS-345 and FWS-398) also revealed a failed condition. 6.2.4.2 Swing Check Valve FWS-378 FWS-378, a 4-inch swing check valve of similar design to FWS-346, is the valve location at which the gasket failed. Figures 6.55 through 6.58 show the valve after disassembly. It did not appear to be damaged Internally as a result of the water hammer, although, the disc seating surface displayed multiple cracks across the sealing surface. The deformed gasket which failed and relieved water hammer pressure loads is shown in Figure 6.55. Figure 6.58 shows one of the elongated bonnet studs which had been stretched about 1/2 inch. 6.2.4.3 Swing Check Valves FWS-438 and FWS-439 FWS-438 and FWS-439 are 12-inch swing check valves located upstream of the i east-side and west-side FW pump, respectively. The failure of FWS-438 to fully ! close is believed to have resulted in the overpressurization and rupture the ! east-side flash evaporator. ! Figure 6.59 111ustrates FWS-438 "as-assembled" and "as found" following post-event examination. The nut had loosened and the disk had rotated the anti-rotation " nub" under the hinge arm, thereby preventing the disc from returning to the full seal position. Figures 6.60 and 6.61 show the topside of the FWS-438 ;
. check disc. Evidently one nub had been subjected to impact for some time (Fig- ;
ure 6.60). There was no provision for a lacking pin in disc nut pin FWS-438. !
, Post-event inspection also revealed a similar failure of the west-side FW check j valve (FWS-439). j 6.3 NRC Fvaluations of Water Hammer i 6.3.1 History and Focus During the early 1970s, the NRC staff became aware of the increasing frequency of water hammer occurrences in nuclear power plant systems and developed con-cerns for the potential challenge to system integrity and operability that 67
l could result from these incidents. For pressurized water reactors, the major contributor to the increased frequency of these incidents was a phenomena j called steam generator water hammer (SGWH) (defined in the opening of this section). The significance of these events varied from plant to plant; how-ever, NRC was concerned that a severe SGWH might develop that would cause a complete loss of feedwater and would, therefore, effect the ability of a plant to cooldown after a reactor trip. r Following the SGWH that occurred at Indian Point Unit 2 in 1972, which resulted in a circumferential weld failure in one of the feedwater lines, NRC sent inquiries and questions to all utilities (including SCE) requesting design and operational information describing design features for avoiding SGWH. In 1978,
- the generic subject of water hammers was declared to be an unresolved safety issue (USI A-1), and therefore received increased NRC and industry attention.
SGWH is associated with the draindown of top feedrings and, therefore, NRC attention was directed at the feedring design, and internal SG components near the FW nozzle. Experience had revealed that internal damage to the feedring and supports could occur. Thus, the modifications implemented to prevent SGWH generally required installation of J-tubes to delay draindown of feedrings, short horizontal runs of FW piping adjacent to the SG feedwater nozzle to mini-mize the magnitude of water hammers, limits on AFW flow rates to avoid too rapid an injection of cold water, etc. In general, attention focused on the internal structure and design of the steam generator rather than on conditions in the FW lines and flow control stations. The NRC was aware of the possibility of developing condensation-induced water hammer extending back into the feedwater piping as a result of line voiding because of a water hammer occurrence at the KRSK0 plant in Yugoslavia in 1979. Limited information on that event suggested that leaky check valves or preoper-ation pump testing (i.e., start and trip tests), or both, were the underlying causes. However, similar occurrences had not been reported for U.S. plants and, apparently check valve failures were not considered a significant contrib-utor to feedwater system water hammer by NRC. The Team conducted interviews of NRC staff involved in the resolution of USI-Al and reviewed staff reports gener-ated by them. The Team did not identify any citable references, decisions or discussions specifically related to why the scope of the staf f's activities associated with the resolution of USI A-1 did not include the potential for and prevention of feedwater water hammers resulting from voiding of main feedwater ifnes due to leaky feedwater check valves. Implicit in the reliance NRC placed on "J" tubes to prevent steam generator feedring voiding, thereby preventing SGWH, was an assumption that feedwater system check valves do not leak. As a - result, the Team conducted additional interviews of NRC staff involved in approv-l ing licensee inservice testing programs for safety-related valves. However, the Team failed to identify citable references, decisions, or discussions which - demonstrated that inservice testing programs for safety-related feedwater system check valves were specifically to be reviewed against criteria that would assure NRC could rely upon the check valve integrity to prevent voiding of feedwater Ifnes. It appears that the NRC did not consider feedwater piping water hammers due to failed check valves to be a substantial contribution to the body of reported occurrences and, therefore, did not pursue this issue further. A compilation of water hammer occurrer.ces, underlying causes, systems affected, and corrective actions taken is contained in NUREG/CR-2509 (Reference 1).
- 6-8 l
NUREG-0918 (Reference 2) summarizes (1) the causes of SGWH, (2) various mea-sures employed to prevent or mitigate SGWH, and (3) the nature and status of modifications that have been made at each operating pressurized water reactor plant. NUREG-0927, Rev. 1 (Reference 3) summarizes technical findings relevant to USI A-1, " Water Hammer," and provides insights into means to minimize or eliminate further water hammer occurrences. NUREG-0993, Rev. 1 (Reference 4) contains the regulatory analysis for USI A-1, and documents public comments received and staff response or actions taken in response to those comments on the resolution for this USI. 6.3.2 SONGS-1 Water Hammer History and Evaluations There have been a number of water hammer occurrences at 50NGS-1, dating back to 1969. Five such occurrences are summarized in NUREG/CR-2509 (Reference 1), three of which involve the feedwater system. The April 29, 1972, January 14, 1974, and May 14, 1979 occurrences damaged FW control valves and piping supports in the main steam and feedwater lines. However, none of these events were attributable by SCE to SGWH, but rather to improper installation of piping supports, inadequate design, faulty flow controller hardware, etc. Correspondence between the NRC and SCE dating back to 1975 (References 5 to 14) clearly indicates NRC and SCE preoccupation with the prevention of steam gener-ator water hammer (SGWH). NRC concluded (Reference 13) that backfitting SONGS-1 steam generators (to install J-tubes on the feedrings) was not warranted because of: (a) existing design features (i.e., the small horizontal length of FW pip-ing at the SG nozzle was not long enough to allow a steam pocket to form), and (b) that SGWH had never occurred at SONGS-1. Since SGWH had never occurred (although it had occurred in other Westinghouse PWRs), the probability of occur-rence was judged to be very low. As a result, NRC also waived the need for estimating SGWH loads on FW piping and supports (Reference 14). Followup inter-views by the Team with NRC staff who were involved in previous 50NGS-1 water hammer reviews supported the above findings that NRC's concerns were with the prevention of SGWH and not gross voiding of the feedwater lines. The potential for cold AFW injection to cause water hammer was also included in the Team's evaluation of the references cited and in followup reviews (Refer-ences 25 to 19). The establishment of an upper AFW flow rate limit (i.e. ,150 gpm per SG), as recommended by Westinghouse (Reference 16), was considered ade-quate and accordingly, operators were alerted in the E01s not to exceed this flow limit whenever the feedring was uncovered to avoid creating SGWH conditions.
. Team members met with SCE staff on December 13, 1985 (Reference 20) and SCE reviewed the SONGS-1 AFW system in terms of the original designs, upgrades (i.e., seismic upgrades), THI Action Plan requirements, etc. These discussions
- also delved into prior water hammer occurrences and evaluations (References 5 to 15). Althuugh this meeting did not uncover any significant new information, it 111ustrated the belief that SGWH would not occur, since it had not occured previously, despite numerous feedring uncovery transients, and that limits on AFW flow rate were sufficient to preclude such an occurrence.
6.4 Valve Inservice festina The ASME Boller and Pressure Vessel Code, Section XI, which specifies valve inservice testing (IST) requirements for these feedwater check valves, states: 6-9 i l
Valves shall be exercised to the position required to fulfill their function unless such operation is not practical during l plant operation . . . Valves that cannot be exercised during plant operation shall be specifically identified by the owner l and shall be full-stroke exercised during cold shutdowns. l Full-stroke exercising during cold shutdowns for all valves not l full-stroke exercised during plant operation shall be on a frequency determined by the intervals between shutdowns as follows: for intervals of 3 months or longer, exercise during each shutdown; for intervals of less than 3 months, full-stroke exercise is not required unless 3 months have passed since last shutdown exercise. - Additionally, the NRC staff position on cold shutdown testing of valves is as follows:
- 1. The ifcensee is to commence testing as soon as the cold shutdown condition is achieved, but not later than 48 hours after shutdown, and continue until complete or the plant is ready to return to power.
l 2. Completion of all valve testing is not a prerequisite to return to ! power.
- 3. Any testing not completed during one cold shutdown should be per-formed during any subsequent cold shutdowns starting from the last test performed at the previous cold shutdown.
In 1977, SCE submitted its IST program to the NRC for review and approval. The program was revised in 1979 and 1983 in response to NRC comments and resubmitted to the NRC in January 1984. The IST program for SONGS 1 has not yet been ap-proved by the NRC (as of January 1986). NRC staff interviewed by the Team at-tribute the delay in approval of the program to (1) the long-term existence of open issues for which SCE did not propose timely resolution and to (2) NRC re-view scheduling problems. SCE implemented the proposed program while awaiting NRC approval. The feedwater system check valves are all periodically tested in the closed position. The main and bypass feedwater regulating check valves are normally tested in cold shutdown (Mode 5) and the feedwater pump discharge check valves are tested in hot standby (Mode 3). The Team reviewed the operating history for Unit 1 for the period from November - 1984 (last refueling outage) to November 1985 to ascertain when IST could have been performed for the feedwater check valves (i.e. , during plant shutdown last- ' l ing more than 48 hours). The periods that IST could have been performed include: . I November 1984 Startup after refueling outage February 9-25, 1985 17 days May 1-11, 1985 10 days August 21-30, 1985 10 days September 19-23, 1985 5 days The IST surveillance records were then reviewed for each of the valves. These records indicate that the valves were last tested as follows: 6-10
Valve Description Last Tested Pass FWS-438 East feedwater pump discharge check November 1984 yes FWS-439 West feedwater pump discharge check November 1984 yes FWS-345 A loop FCV-456 check February 1985 yes FWS-346 8 loop FCV-457 check February 1985 yes FWS-398 C loop FCV-458 check February 1985 yes FWS-379 A loop CV-142 check October 1984 yes FWS-378 8 loop CV-144 check October 1984 yes FWS-417 C loop CV-143 check October 1984 yes ~ There are 121 valves that ace subject to IST during cold shutdown. Although IST was performed during each outage, all of the valves were not tested. Consequently, the feedwater valves were tested only one time since October 1984. The available opportunities for valve IST were not always fully utilized due to higher priority operational requirements. Surveillance test procedures for verification of check valve closure for the main feedwater pump discharge valves (FWS-438 and FWS-439) require one main feedwater pump to be running while the other pump is stopped. The discharge valve at the idle pump is then opened and the pressure is monitored between the pump and its discharge check valve. An increase in pressure or an operator observation that the pump is rotating backwards would indicate that the check valve is not closed. While providing reasonable assurance of check valve closure, this testing method also subjects the low pressure pump suction piping to some relatively high discharge pressures if the check valve failed to close (as in the November 1985 event) and thus damage is possible to such components as the flash evaporator. Testing with the idle pump suction valve shut would provide a more rigorous test. Surveillance test procedures for verifying that the other main feedwater check valves are closed require testing to be performed during plant cold shutdown with the steam generators filled to above the feedring. The motor-operated valve upstream of the check valves is closed and the drain valve between this valve and the associated check valve is opened. The column of water in the steam generator provides approximately 4.5 psi differential pressure across the valve to provide the closing force on the check valve disc. The procedure states that this section of piping is to be drained, and that little or no flow from the drain should be verified. This test procedure leaves the surveillance operator to make the decision about how much flow is "little" and thus indica-tive of positive verification of check valve closure. The IST records do not provide a means of determining if flow occurred or its extent, or for verifying that complete valve cavity drainage occurred before a determination was made
. that "little, or no flow" occurred.
Valves FWS-345 and FWS-346 failed the IST on February 24, 1985 when tested dur-ing Mode 5 (cold shutdown). Maintenance work orders were prepared to repair both valves. However, on February 26, 1985, a "Non-routine and Increased Fre-quency IST" was performed during Mode J (hot standby), and the valves passed. During Mode 3 the steam generator pressure increased the force available to seat the check valvos (to approximately 700 psia) and thereby have enabled them to pass. Testing at the higher differential pressure provides a more rigorous test. The work orders were then cancelled and no corrective maintenance was performed. 6-11
i 6.5 Feedwater System Check Valve Maintenance The main feedwater pump discharge check valves (FWS-438 and FWS-439) were last ' disassembled, inspected and reassembled on May 5, 1980. All internals and
- seating surfaces were reported to be in good condition. No additional recorded maintenance since that time has been found. ,
i Maintenance on the main feedwater regulator check valves (FWS-345, FWS-346, FWS-398) dates back to the SONGS-1 refueling outage in 1975. The maintenance history can be summarized as follows: < 1975 Refueling:: Inspected three main feedwater regulator check valves and installed new internals in all three check valves. 1977 Refueling: Inspected and installed new internals in all three check valves. ( 1978 Refueling: Inspected all three check valves, found no problems, !
; cleaned parts, and reassembled valves.
1980 Refueling: Inspected all three check valves, replaced cotter pins, washers and nuts on flappers, and reassembled. i i SCE personnel could not identify any other completed maintenance work orders for i any of the check valves in the feedwater system. However, a cancelled work ' order was found for FWS-345 (M0 #84103233000) and FWS-346 (M0 #84103232000), the feedwater regulatory valve check valves for steam generators A and 8, re- i spectively. These work orders originated about October 26, 1984 because each valve had failed IST according to the problem stated on the work order. The work orders were cancelled on October 28, 1984, although the reasons for their i i cancellation were not stated. Further, a review of the completed surveillance ; test results for October 1984 indicated that both valves passed. The Team was : not able to identify the reasons for these apparent discrepancies, i r A 1976 SCE incident report summarized a failure of the 8 feedwater check i valve (FWS-346) that was identified by an unusual noise near the feedwater i regulator valve (similar to the noise in June 1985 discussed below). When the ' valve was inspected, workers found that the disc had fallen to the bottom of the l' valve body (as happened during this event). These inspection activities would i have been authorized by a maintenance work order, but SCE could not retrieve a , ; related work order. 6.6 Feedwater Train Noise Investiaation .
, On June 24, 1985 (Reference 21), a rapping noise was noticed in the breezeway between the containment sphere and the turbine building. Initial investigation i with the aide of a stethoscope indicated that the rapping metallic sound i appeared to originate from the vicinity of feedwater block valve FWS-342, just !
downstream of the 10-inch feedwater check valve FWS-346. The block valve was radiographed and the valve manufacturer contacted in an attempt to determine the source of the noise. In a subsequent evaluation (Reference 21), the pos- j sible causes of the noise were believed to be: (1) a loose hinge pin in check i valve FWS-346; (2) a loose disc is check valve FWS-346; (3) a rattiIng disc in l l block valve FWS 342; and, cavitation originating in the area of the check or j block valve. Although the exact cause was not determined, SCE concluded that > 2 the possible causes did not threaten plant safety and scheduled further testing. l c 6-12 f l
A presentation was made to the Onsite Review Committee (OSRC) on July 18, 1985. The presentation is based on an earlier memorandum which summarizes the investigation and concludes: Station Technical feels that it is safe to continue to operate the Unit since the failure of either valve (block or check) does not decrease the margin of safety of the plant. The components of the valve are large enough to pose only a very remote chance of traveling to and damaging the Steam Generator tubes. ~ The OSRC concurred with this conclusion. On September 3, 1985, SCE's Nuclear Safety Group reviewed the minutes of the Committee meeting and concluded that the station was taking appropriate action related to the 8 feedwater train noise. ' Maintenance orders were generated to inspect the block and check valves during the next available opportunity, which occurred in October 1985 when the plant was shut down for another reason. However, it was decided to delay the inspec-tion of the valves until the refueling outage scheduled to begin at the end of November 1985, because no variation in the feedwater noise sound level or fre-quency had been noted. 6.7 Valve Failure-Related Findinas Check valve failures caused by partial disassembly while in service do not ap-pear to be unique to SONGS-1 or to the valve manufacturer (MCC Pacific). A Ilmited review of Licensee Event Reports (LER) indicates that these valve fail-ures are not unique. The Team reviewed 33 LERs that identified check valve failures in feNwater systems; these indicated that 11 check valves failed be-cause the d N. failed to close or because the disc retaining nuts, studs or locking pins failed to allow proper operation of the check valves, resulting in system backflow. None of these LERs speciffed that the failed check valves were manufactured by MCC Pactfic, who was the supplier of the failed check valves at SONGS-1. Failure of FWS-438 and FWS-439, the main feedwater pump discharge check valves, may have been due to inadequate valve design, since the disc-retaining nut was not provided with a positive locking device that should have reduced the pro-bability of the disc working loose and wedging into the valve seat and falling open. Additionally, excessive clearances between the hinge and disc assembly allowed the disc to rotate past the anti-rotation devices. The failure of FWS-346, the 8 feedwater header check valve, may have been caused by the inadequate hardness of the disc-attaching stud, which allowed the threads to strip and the end to mushroom over, conditions contributing to the ultimate valve failure. However, the service conditions (f.e., flow-induced vibration) experienced by this valve may also be a major contributor to failure. Such vibration was suspected during the June, 1985, investigation, and as indicated by the significantly mushroomed end of the valve disc attaching stud. Failure of FWS-345 and FWS-398, the A and C feedwater regulator check valves, may have been due to similar service conditions. The cracks in the seating surface of FWS-378, the 4 inch check valve in the B . loop bypass line Appear to be service related. However, these cracks may be due to the significant forces on the valve from the water hammer. G 13
Failure of the yoke of FCV-457, the 8 feedwater regulating valve, was probably due to lack of sufficient support or bracing of the valve operator during the pipe movement when water hammer loading occurred. MCC Pacific Valve Co. Bulletin 400 (1978) cautions owners to check the opera-tional environment and to avoid conditions that lead to high turbulence which can damage valve internal parts and shorten valve life. This same Bulletin t also states (under " Service Recommendations") that: Service in systems involving rapid and frequent flow reversals, pulsation or excessively turbulent flow should be avoided. - Locating swing check valves away from elbows, equipment, etc. within the piping system can often minimize or eliminate problems caused by this type of application. Suspected pro- . blem systems should be reviewed with the valve manufacturer before selecting and purchasing swing check valves. The 1975 SONGS-1 refueling maintenance records for the MFW check valves stated: Main feedwater check valves internal discs, hinge arms and hinge pins were replaced. Excessive wear was noted on the hinge pins and hinge arms. Despite this finding, the Team did not find evidence that the valve manufacturer was contacted as a followup to any previous check valve deteriorations. Rather, the internals were periodically replaced and operation resumed. Spinning of discs and induced damage by turbulence in swing check valves are not a new phenomenon. An article in Power, February 1983, states: Spinning of the disc by fluid forces has injured many swing check valves. Sometimes the disc stem has worn completely through, allowing the disc to float downstream. Anti-rotation pins (Fig. 202, p S-42) can prevent this. This is the type of valve failure discussed in Section 6.3.4. These IST procedures do not provide a positive means to detect a damaged, or degraded check valve. Only periodic disassembly, inspection and maintenance can provide such assurance. 9 1 0-14
References
- 1. R. L. Chapman, et al, " Compilation of Data Concerning Known and Suspected Water Hammer Events in Nuclear Power Plants, CY1969-May 1981,"
NUREG/CR-2059, May 1982.
- 2. Anderson, N. and Han, J. T. , " Prevention and Mitigation of Steam Generator Water Hamer Events in PWR Plants," NUREG-0918, November 1982.
- 3. Serkiz, A. W. , " Evaluation of Water Hamer Occurrence in Nuclear Power Plants, Technical Training Relevant to USI A-1," NUREG-0927, Rev. 1, March, 1984.
- 4. Serkiz, A. W. , " Regulatory Analysis for USI A-1, " Water Hamer,"
NUREG-0993, Rev. 1, March 1984.
- 5. K. P. Baskin, Southern California Edison Company (SCEC) letter to R. A.
Purple, NRC,
Subject:
"Secondiry System Fluid Flow Instability,"
July 14, 1975.
- 6. K. P. Baskin, SCEC, letter to A. Schwencer, NRC,
Subject:
" Susceptibility to Steam Generator Feedwater Line Water Hammer Events," De'.iber 27, 1977. -
- 7. J. 8. Moore, SCEC, letter to R. H. Engelken, NRC-R/V,
Subject:
" Degraded Conditions of Hydraulic Snubbers," November 6, 1978.
- 8. K. P. Baskin, SCEC, letter to D. L. Ziemann, NRC,
Subject:
"Information Requested on Feedwater Lines," July 3, 1979.
- 9. J. G. Haynes, SCEC, letter to 0. L. Zefmann, NRC,
Subject:
"Information Requested Concerning Steam Generator Water Hammer," August 31, 1979.
- 10. D. L. Ziemenn, NRC, letter to J. H. Drake, SCEC,
Subject:
" Steam Generator Water Hammer," September 12, 1979.
- 11. K. P. Baskf n, SCEC, letter to D. L. Ziemann, NRC,
Subject:
" Steam Generator Water Hamer," February 14, 1980.
- 12. J. A.
Dearf en,
EG&G Idaho, letter to R. E. Tiller, 00E,
Subject:
" Steam Generator Water Hammer Technical Evaluation, San Onofre Unit 1," March 31, 1980,
- 13. D. L. Ziemann, NRC, transmittal to R. Oletch, SCEC,
Subject:
" Steam a
Generator Water Hammer," April 22, 1980,
- 14. J. G. Haynes, SCEC, letter to D. L. Ziemann, NRC,
Subject:
" Steam Generator Water Hammer San Onofre Nuclear Generating Station Unit 1,"
April 22, 1980.
- 15. K. P. Baskin, SCEC, transmittal to D. M. Crutchfield, NRC,
Subject:
" Automatic Initiation of Auxiliary Feedwater System SONGS-1," Harch 6, 1981.
6-15
- 16. Technical Bulletin 75-7, " Water Hammer in Steam Generator Feedwater Lines," Westinghouse Nuclear Services Division, March 9, 1977, Exhibit 89-1,
- 17. D. G. Eisenhut, NRC, to J. H. Drake, SCEC,
Subject:
" Auxiliary Feedwater System Flow Requirements," November 15, 1979.
- 18. D. M. Crutchfield, NRR, to R. Dietch, SCEC, " Auxiliary Feedwater System Automatic Initiation and Flow Indications (TMI Action Plan Item II.E.1.2),
Novemoer 18, 1982.
- 19. W. A. Paulson, NRC to K. P. Baskin, SCEC, " Auxiliary Feedwater System .
Technical Sepcifications," October 21, 1984. \
- 20. Transcript (Set 3-044), December 13, 1985 Meeting at Rosemead, California.
- 21. Exhibit No. 355 B Feedwater Noise Investigation, provided by SCE on December 11, 1985.
4 6-16 L
Table 6.1 Description and Corresponding Illustrations of Feedwater Pipe Damage Following SONGS-1 Water Hammer Description of Component, Support Figure Damage, Motion, Etc. Location (s)* 6.9 This snubber station, the closest to the H00A SG 8, showed no visible damage or pipe H008 movement. The feedwater pipe turns H00C vertically, and at an angle, to rise approximately 10 feet to mate with
, the SG feedwater inlet nozzle.
6.10&6.11 These support stations were the first H000 that showed damage (or movement) caused H005 by water hammer. H006 6.12 View of FW pipe elbow at northeast corner Downstream showing dent in pipe that resulted when the H006 pipe hitting the concrete corner and then rebounding. 6.13 View of pipe (looking souch) showing H00G movement of approximately 12 inches, slippage of vertical support pads off channel beam structures and downward drop of FW pipe. 6.14 View of support HOOG looking in opposite HOOG direction from Figure 6.13. 6.15 View of horizontal and vertical support H00H pads displaced southward approxiately 12 inches. 6.16 Evidence of first lateral motion (east- 120 ward); note deformed vertical structure, and then axial rebounding which displaced pipe supports approximately 12 inches south-ward. 6.17 Close up of scratch marks which show 120 evidence of water hammer forces driving pipe inward and then displacing it northward. 6.18 Further evidence of axial displacement of H00J FW pipe on east side wall looking north towards support H00J.
*See Figure 6.7 for support locations and identification.
6-17
Table 6.1 Description and Corresponding Illustrations of Feedwater Pipe Damage Following SONGS-1 Water Hammer (continued) Description of Component, Support Figure Damage, Motion, Etc. Location (s)* 6.19, 6.20 A series of photos illustrating damage HOOK 6.21, 6.22 incurred at the support structure down-6.23, 6.24 stream of the southeast elbow. The damage incurred by the structure clearly illustrates - the magnitude of pipe motion which occurred during the water hammer pulse. 6.25 Permanent set (i.e., bend) in FW at elbow pipe. View at elbow from support near sup-HOOK and looking west toward support ports HOOK HOOL. Pipe has been bent laterally and HOOL south from support HOOL to SE corner elbow. 6.26 View showing lateral movement (westward) HOOL of pipe which resulted in sheared vertical support structure. 6.27 View of spalled concrete and support plate HOOL damaged by water hammer, nuts were loosened and bolts were missing in wall plates. 6.28 View of piping and support damage just H00M downstream of where FW 8 line takes a 90* bend to exit the containment building. Note: (a) bent white stucture showing evidence of eastward motion, (b) vertical displacement of pipe until restrained vertically by white structure. 6.29 A wide angle view of supports H00M and H00M H014. Snubber at H014 was bent westward H014 and FW piping was driven upward. 6.30 Shows vertical support bending caused by H015 vertical motion of FW piping just down-stream of containment penetration C-3G.
- See Figure 6.7 for support locations and identification, i
l l l 6-18
If T CONDENSATION SURFACE
+ + STEAM + %
- a. Stop Valve Has Closed and Reflit Starts
{0dD if T ( REPLENISHMENT OF CONDENSED STEAM 4: 's t-I
- b. Cold Water Has Filled Bottom of Pipe - AFW
,o
- c. Pipe is Nearly Full and Surface Waves Form - AFW If stem voes
- d. Slug Flow Conditions are Established -A FW Figure 6.1 Filling of a Voided Feedwater Line
JL 1 -
~
=si I
= 18$ a eg eh a N '
g " " X I !. s
-5 m -5 I I I
E e RE
-8 e
e n d i- n g WL 2 a QL m 5 Eb 3
!b g 3 o-X s o-X !.
,, ,, x kN kN $
it n
. m kN g kN ! O m
d 5 d b ! E $ E
.e m
sJ a d 5 e
~
is:
- , ist
- a .
3 I vmmea w i 6 "
====
l
; SG "C" CONTA0LS O SG *C" M
FAILED GASMET fr STRETCHED BOLTS SYPASStmE 372
.144 DMG v_ _
378 CONCRETE TUR8MIE WALL YDKE CLOSURE CISK FW CONTROLS FOUND IN BOTTOM / OF VALVE CAVITY / 1I
,' y X # -- ,e + (SAAAE CONCEPT AS FOR ~Bh adov fcv rWS # d i #;
29 457 342 /j q
>< N V,WS FWS 308 FWS 386 k\ 3g4 2"WENTtWE E ~I M
FLOW ? SG "A" CONTROLS Y 7 SG *A* FROIA FEIDWATER TR 4!d Figure 6.3 SONGS-1 Loop "B" Steam Generator Flow Control Station e
A A
=
G G s s y = > . w4 l2 A3 Nu3 w22 mC V00 F3 3
*v1c3B t3 :
I ' m l ' t e s y w00 S l A3 w0 1
- )
Au3 W F Ar E ( r t e
, a w02 N3 7
1 2 Nu3 d w e e F y r i a l i x u A M (h os v2 s2 a1 1 S G
" N Ol O
w Aa i " S 4 a 6 y e r u ig - F . 0* s i e 1 i G G* _ g w u w u V y . \'/ c wm f A i
'8
9
,, , PM EgEEDUJ i\TE FZ. o TC MP '
.Q.;;.NTf,$ L.( h 5 Q '. .
&'^f ~, , ,,, , ; i . . 1 M n 'mov s i nes)
'. h [ '
b '
'N . . ', hh e h, '- * -
~[
'( A h. .
' ,* - 4
_ ',_ h ji f, ,-y, i s j ' '; '
'r '
E' '!' ' h 1 l, J I i l .L;ul .l i ' i
^:
!jI lII ,ij h 3 GSO-ll .i I 300l9M I 325 I l 1 375{~ l 400l 4a5' l
l li.l b'EGd FIHft.
-- ----s ~___ .l. II' 1 :l! - -
c , l j- . l ( 7 l~'[2,. -
$ { -( - - "
1
$/dk \-% [ '
Jp 3 3'-__- y ,
-r-' ~n
.r
..,,,.......r"r. r:~ r--~~'~ ~
t--:-r-----"- - u ,h,FW LOOP "B", ISOLATED .@ 06:30 a ei , g 7Au / , _
/ i alu. i, . o ,L ll L. I
) -l
;;'.rs-~' ' /- --
COOLING CONTINUES < l g 81 ll l . J jj{ - 1 gg (-{ 1
- 1) i.-.. ___. _ _ c::: WATER HAMMER REPORTED @ 05:07 s r" l L
/ li 300 lM / 335 360 I Ill '
il.425 375 400 1
/
/ [ .lllll. 1,1 l l; P.E.Gb".". .
ll j -OPERATORS ISO. LATE VIA MOV-20 1 j N' l f . p II ,... -- RAPID COOL DOWN DUE TO
. __ ,y ,, -_..- ___ .-
/.. ._.
.~
l l 5 )U4. Ill! !!!FT '-
% ) .
7 I i l
~ BACKFLOW OF AFW @ 150 gpm IllllllllillIllillll'lli :l .
!! f,i l
l' i s.,
! t l lI l l~
'll/~' !l 0! l ,l!
a tra o o itc r.- _!,,,,.1 M " dl:I ll il "
/ i i 4-MIN LOGS OF RECORDER ti '
fM'i i I
~T i . ,r di i
- X ll llll l i1 i
i i 11.1, TRANSIENT COMMENCES AT 04:51 i e i, I !i,m i POWER l ll t fp i e-
' ,; ,;!I lll
('
'I l j r; . -
_L1Li l ! l i
- l. l i l t lll ; ;{i It I i i
!! I I' li i ,-
l i__ L1 it!l !lt-' W - Figure 6.Sa Temperature Recorder TR-45S
l
- --e
< l
- m 9 4 b- l 9 -
w a , w .s ~ / .f. 1 e- r- % / A..w - N 1,, './ c,d s 7( 7 s,,
.m b 'k - ' -
- J x's i 3 .
. e. .-ns .' N \ p _+-_- s
&. m
- w ~ % '*.~,x 'sd s,
.N s
's , g.
1 X,r ' ., I gg \i ... ,h.* N , . .,'s, s, .s -
'. , I l
, f ,, , , , g , .
.s .
y\
~~ . . . ;%,- g .N s,
^'
w a' -' '
%rg ~- '
,- i w . b3. /'- /
.mo ' ,'
e' ,
, '300 , j
-; s - A, . ,.;'\s, :
9 a, 3 )/ r g, ',* , '
'/. ,- gr
~
,J 'd .
,- N 'N -, f
'b " 'N[
UI3 n a. us 5t ./ ;.' ', -
,/
. i
\
y. 300' .
--r(. n j.
,x s/ - ' [ ,. f.
w
< A ,/ /- -,%
e ," ,-, , . .
- .. / C) ,ys x
fg
*h n . .. ys.n x u.
/
/
, ./ p s f' . ',s# , h .....A-l ,
. \$ \ t~'
7
'f ~ s~.
./
a O n.- o x,f.. ,
.7'7',
i U' Si p_ '," - j,
,y a
]- t@[,, 7 -l2w . ?x,
ts lx,'s'y .
/ &.:.d
~.y g /~.\ .'X.
/ lf ' f f., , .-@
\ L, . ~
p/ 'd
. i . . -
Y l Vi s
',$.i g , ; d\ l"-l. l', ; , A?'%' ' ,-N-3' o\ d /N ',S, S ',' \
*',.*',.'.v'--'2 ' 8l '. 'y'
. '\ .
r-gi: y, 4 yeQ j ; ;a. . \ i f --l3. -7. i{. .!. mhr ._ ,->-gps,; g . .. l ;
*--T \1 n,,.- to'
' 'f T' U
=
g t , . . .
\. , - u id, . A
. d . c , . i. . ,3 <. . . g, p l . . . .
i a s " :.li ./ . ,. .,
,L t _ _+ -
,Q g 'i - ..
m.
; ;t t \ .\ ,., -\'
. a>% . A gG ,%.: i .. '
<a g a.
g.4' . \\, i Q.t P ,' t \
,' , - . 9 ~ e! / : e
\. m '\. . \. ', ,
sh
& / s, ' i j -i ..1 . q,:/ '
\,k, ,\
~
~~
, A, . 'x.'gg g s ', . ,, - / '
I w w , r, . ~. x . t - r " - .../- I,g bw 4
.* \ ..g s N,.r .3 3 -s .y', s' e < -
*' . - l's qs ,
g .
. \. ',. r, p.
g - .\ '*r. .. . ,N-- *'. s' \._x/' ' 5- f k\ , . i s ., '.\.,k.s\ 's/ [.
/
- q . s
- c. . .
' ' .i .. .
s ,
\<, \ j .: i.,
~ '.3. . 4" , '. ',A\' .. ;-
g t ,N
,* ' 4-. e j
jN, , " 4,f . . \y .. T<\ ,,. ' .,\. .x* f,
,j. \ ,s s. s ; g; 1 % - ,. .._ .
.s c- .s ,
' , A,, .
's
'- i
,1 , ,
L_~.-_-. 9,__- W.~; /,'\{y.', j ,_
,< N '
\
,, .\ ~.ly.,/aLK'
.. ,. k s -2.:. 4. s .
. . . . ., ' . v .
, \ %. s .I 7.' s ~- ,
\ '\ Rl%:' s,f~ . 3%. g A gg ;. '
^
N . 4.,/s'. \.%, ,s!'s...
- . . . . s ..
. [ ~ - h%.- ~~'*)s '.'. .
- s. :'
3' N
.s ' ' - -.M , ..' ,.
1
#%N N'
. , f . .N.( --- ," . .N, - ,- , . , l ]
,.'8 - .
.'N. . .- . .- ..-..
,'* - N- %,.. , . . , - - -
..\_-~ .
'/,
L N- ' . ?y* ~ E- - b - ', s\; - T "N'Q -+,--
- ' 'e
'f-j]'. ,
L, N9 - s L ..
-c
- s. . - "
SLOW REFILLING
. OF HORIZONTAL PIPE
- a. Loop "B" Pipe 90% Full When AFW Stopped BACKFLOW OF STEAM TO REPLACE STEAM BEING CONDENSED CRITICAL FLOW CONDITIONS
- b. Wave instabilities Being Formed Prior to Bridging of the Vertical Elbow
- c. Elbow Has Bridged and Vertical Lag Filled
?
STEAM POCKET
,e
.g :- .
- d. Probabis Steam Pocket Location Just Prior to Loop "B" Water tiammer j 1
Figure 6.6 Refilling of Loop "B" Leading to Water Hammer l
O
\\
R a= .
'** \
/ 4
\ /
+
, \
es s s y $
~ 4 5 4 .-
h
+
4 D ,
.4
+
h
- / c ja
't, a
?
3 h 5 l/ / c i i
\ / p, '
x
\. \,
Y,
/
\
- Y-
\, ,,
[k c
/
\
\
p\,
$\
\
1 1 i D
~
. *} n J
a t 1. il 5! li A o 5"5 3
/\"nIII g- 12 ,N[ <
/!
* ((g* j!ii r 5 / / \* I
!i ( /
\
b 8
- 31
* \/ /
\
/ j ii , - - </ ff
\
\ / g i v f
\ l 0 8Il\
/ = / 8
/
/ s gE* V E a
e gS ._ Na e - h 5 El I'! / o
/ 3 i
\ - l"
! + \ l N gjiji h
\
a n!
\ .. I J E I l
. - - - .. --- - -.. . .-.. ~ ____.-. - __ - .-.._ .. __._-. ._ - .
I 1 l i i I l i l , l VERTICAL /SEMlVERTICAL RUN OF FW LINE "B" TO SG "B" LOOKING WESTWARD.* I i j 1 SUPPORTS HOOA. HOOB.
& HOOC PROVIDE A RIGID
( SUPPORT. NO DAMAGE WAS i OBSERVED AT THIS SUPPORT STATION. 1 1 i I l l J Figure 6.9 FW Line "B" Support Station
- Note. See Figure 6.7 for Direction Orientation.
) i
l l
,, NOTE LATERAL I E WES ARD
- ~ * *
#* r* 8 NCHE AN DISPLACEMENT OF PIPE INWARD l
- , O RDS CONCRETE 1
l Figure 6.10 FW Line "B" Loop Support HOOD t
~
l l
.,- NOTE BENT BRACKET
, ,, i ' ' DUE TO LATERAL PlPE MOTION. VIEW i
LOOKS WESTWARD TOWARDS SUPPORTS HOOA.HOOB.6HOOC l gure 6.11 lFW Line 'B' Loop Supports H005 & H006 l
'yQ(
IW" l 1
_____.____. _ __.. ~ - _ _. _.. _ ___.__________ _ . _ - ._ _ _ _ ____ _ _ _ __ _ CONCRETE CORNER l l 4 LOOKING TOWARDS SG B" DENT IN FW PIPE -
- p.
E w ry; I .
- CORNER ELBCW WHICH LN.' .? !U,F ' ,f 4 BULGED APPROXIMATELY Y; * ~ . .4 ; ,
- l
%lNCH t7[gl l
Figure 6.12 ' Pipe Downstream From H006 Support .i
- ;$ f'(,~,
*.s a.
l j .dh $E ,*.
- r. ~ . ,
[ ;_ . . fe.. ar.i : gf) ' t, .I BULGED ELBOW l . . k.. ? I
~'
i M . , y ' '. - , j ,
$ ._ ~-
, NOTE PREVIOUS SUPPORT -
l ' ' ~ POSITION: PIPE WAS MOVED AXlALLY SOUTHWARD
*W 3
.- % i t Figure 6.13 FW Line "B" Loop Support HOOG
l l y-
;. . + fy,.1'
-c ' . \ g ,, . . $..
9...
. . .g. y: ;err.,
., .r , ,
_jh. .Oaf .: .
- f '. '
f& ;'"~ ~ I : .. 1 - . .. ~. ... h, ..".; . J - T. ; " '
..;... ,,_ .i . = ,
I* **
..w ;
, ' .. ;. ? -
- e. - '
- h. .
' w~ ,
,,3 3r i
.~
v
~
. - y y. : --
- c_ .
4.- y.. l -
. _ . . . . . . . . . . . . . . . t l
l Figure 6.14 FW Line "B" Loop Support HOOG, Looking Northward l ! i
- l
[. ,
/ ,' PIPE SUPPORT PADS f ,
. ,.py -
t. 9
. 4 ,
k g((. .4 . ?i'e'-[' +p,, ' ' . ?.NOTE - DEFORMED VERTICAL l p,"4.d JgU CHANNEL DUE TO IN BOARD j fk , (WESTWARD) PIPE IMPACT t DUE TO ' VATER HAMMER i 5
.e-k i -
Figure 6.15 FW Line "B" Loop Support HOOH l 1 l l
t l l M/ MQF3, l ! T- 4 I;; 'l I
- x n
~
. _ i.u. ; ,- '
w m.% . ~ l f.1* s.tfy bh.w .
; ~) c. "f ., .
I L, Q, l *'
. ' ,$c ,, ! e ..
l ' s.? i - b
'A 1 ?'
.I Figure 6.16 FW Line "B" Loop Support Point 120 (Which is 20'-6" South of Support HOOH) i i
l e l ,
- , e 1 '
.". J .",<. -
j A.,. : . J . l l
*?
l SCR ATCH M ARKS SHO(V i . EVIDENCE OF INITIAL WATER
, i H AMMER LOADS DRIVING Ft*,
LINE INWARD (EAST) AND LATERAL ISOUTH) FOLLOWING REBOUNDING AT THE j , DOWNSTREAM ELBOW , 1 2 ,T l 1 E a
- 3. y .
u oO 4
..e
.3 . .i a 1
Figure 6.17 FW Line "B" Loop Support Point 120 f i I _ _ _ _ _ _ _ . _ . _-
l l l l l l l 1 l l l
*'( , #I , _y G. i (h _ *:, *, b ' [' .
.h -!' * ! .? , l l 5 h ? l" . ., ,
e, r .-. , '.
.. ' l
~'
[ . . . < ,- . {gg 4 '
; . . . . . ,. FW LIN E C' p ..
-,' '~ ' :3 s .
jfQ-p,~
- 4 .' ,
., ~ .,
l
+p, s.
2 v u .
\
l , ,-. .
$" +
- s!
n HOOJ
.,' [,1 M % > ~.
}
( ,, s 4, s.. y r*
- 8f. , .
- h -
/
u .. -- v g 9 n
< e
.; , y. .
ty e -~ y . ,,
' ~
f . l.
,; . l.
.l .
[ Figure 6.18 FW Line 'B' Loop Support HOOJ
. . . . . . . . . . . . . . . .p. ., : ' . . ., g ,
3
. ame. . . . _
.k' '
' S h, r.
- h. .
?%$'Yb
". - N - s.' \'
- I i s A . . , . ,, .5 s
4 g v- - s
- g. j. ' ;L '
VIEW LOOKING SOUTH FW LI N E
' l l H AS MOVED INWARD 4g .'.y ' & UPWARD
,,y 4 . . .
g ,. e 3:.8
.!I R :
-)
. 4, , l , y . - -.f-
- 2-
+- *
.', ; og . ,
s.. .. . ,: .* ..
- m. -
n_, .. l Figure 6.19 FW Line B' Loop Support Station HOOK 1
_,.sm meaa o m um---am---w w--w -m- w-H_m w m--m 'emmm--W 'm__A,n_M a-wku-t O m 5 . m w- v z 2 l o o- = 1 r; = m *
-.. 4" < o COS>
awa:
=y 0
j s CDWC, C,0 AC$u~ J~ I o w 7. 8 5 - y ES<_c2 N sE5ES
- o352 !
! BrrEs strs3 f i l i i ) 2 o m . l 4 o S E?
~
s, ,. y D h _... ~ ' -Q \ ' z O - 2 V
=
5 3 m _. m< O N 3- @ d=
>9 wg 2
9: .9 az u.
I l l l CLOSEUP VIEW LOOKING DOWNW ARD AT HOOK 1 Figure 6.22 Closeup of Damage at Support Station HOOK
,g.,.g i - *J .. 4sc ._,. , e$ j. .. .
.. g
*pl * ~ . '4
~
+ s. , ,;
..,1 -
^
. 'g
+
3
. .. g . -
i
' .' 47.-
..me; : : ?
, %f * '
', wy.
, y .. t
... ,v, . s *qg ,. - -
.( 4 f w.; ,,
-4w $ -
. ,y . n:; - -., ,' s ; -.
en*> * +.
.., 7-A . t.- - ..;..,. .
.J-- - n_
- v. .
f.., , q' ,i
..4,, ,. , . . . ., ..
.N . a-
..h; % *..-
... ' * ~
Fiqur+ 6 23 Torn Pipe. Suppiir t S t ru c t u r e- HOOK
l l l l l l i j . l / '
- y
/ l _..~- i 1
~ -
l a. a n'ac. m N - / M::[%A;%y.w
,zm ' ' 29,..::.1:
';f: J ,
l l l l Figure 6.24 Torn Pipe Support Structure HOOK l i
!' I l
-a j
.* ;, c
... . i 1
j ?
,s. _-W.
l 1
. . . , 7,- *- p j
{
,i :
, -{
.. , .,~
s. e. l
-, . .,s :
~. . ..
.. c .. . ,-.
eI _... *'
?
' .; 1a
, L,y ,.
- s.; .
l j- .! 1 ,: .. . j
-0
! , s- -
+
;, 5,
/, -
- K <-1 -
; ~., : a, * +
, p' . .
n:,: .
.a ;. .
. .w-f... s -
o.,. - s i, 3 ?, ,.
., ( --
., g, , !
_ - g,
, .5 : - e a ^- ,. . . j .a. ,. ..
1 Figure 6.25 Outward (or Lateral) Set in FW Line "B" Looking i l West Toward Support HOOL
)
i I i I
... A "
l SHEARED VERTICAL SUPPORT l l SPALLED CONCRETE
- . / .
g-- - .. l , 1 C I . I j y .;t.
,. s. . . c . .;j
- E_%.de '
Figure 6.26 FW Line "B'* Sheared Support at Support HOOL l MISSING ANCHOR BOLT l l l
. . . ~
\ LOOSENED BOLTS
;~ . . . -
.q r O 3I ,,. .
I,W,'..- .
~28 ,. s.e:: . ...
Figure 6.27 FW Line "B" Wall Support Structure at Support HOOL l l l
t i I i ! ELDOW
' _. 7 4./~.l ? & ' ' +
^
l ^ ci _r --
% . ' ~; . ,. ,
.r .I y. J l
- + - - ^> c . d, ' , -2*. 7.41z.y$ r p, .- N
- ..g; NOTE SUPPORT STRUCTURE j .. .
. -;.- ,5 BENT EASTWARD AND
, ; , .. . FW PIPE BEING I
-, ' A *WP M - -
RESTRAINED FROM
",3 . . . . . . . -
' ' 6; -.
l * . s. ,3
.:. . ; FURTHER VERTICAL i A> s. . --. , , . . , g.,g , ']:^??'.[" MOVEMENT BY SUPPORT HOOM .
i - K -
% ,f, y ..-; C ' (AT TOP).
g ; :r ,c. \ l Y.y _j ,..-i .
- t. . .. g , ._.g
, s 'q ;. -
t
.e -
. s
\ (s&/ .
. * *7% h. ' &.. .l' I
, l , t '~' i --lN , '. r,L',. -
, ..' h,h -
I Figure 6.28a FW Line "B" Loop Support at Support ;HOOM i t i i _ ' .\; ' '
'}_- _
- r. .
.. ,g t .
. - - . p . +-
.n.. .. - .e Y
L .. v. ;
'ff
'M J. ?,y...
.'^ . . z 1); .
AJ .4 4 . ' .' . ]. ,:
, ,j /. f h'
. .. ' / . ' < . ,R l . ' ';
i Figure 6.28b FW Line "B" at Support H014 and HOOM i 1
k ! I i i
- a (
- i,
_N ' 4 l t I ! i
! NOTE BENT VERTICAL L I
ROD DUE TO UPWARD
-- - MOTION OF PIPE f
I I ! I I \ j -9 j [. ' . l I i Figure 6.29 Pipe Hanger Damage at Support Station H015 <
! t s
l l
m , -- k f4 1
~ '\
M s g
' \f I e+
\ ;
0, i e f 0 1 , s.
'N~g <
a . j O b w
\
d I a e o :
\. e x !*
le=< gN 5
~
3 8, X, ,
/
/
rl NOTICE CIRCUMFERENTIAL i CRACK IN CONCRETE i T*7 a TURBINE BUILDING WALL
. . . t. '
. .. 4 .4 q c,. .y
. vy,;
{ ^* W I e 6,, . ,- t
~ ' ' ~
. E'Y.~Y
;b
,1'd t D 4
Figure 6.31 FW Loop "B Penetration at Turbine Building Wall I I i ! Ig 1
.. \
l l.T ... . . i l
/ \
l l l
- f. I
\ l ' l s,': i s l , l . - t l l FWS.342 FWS M6 I i Figure 6.32 Inspection of FW Valving l
%s 4.
g. h4 ,. ;,-
,m.
.( -
I! ; ' T ,
= . .~-
. r-
.g.
c .
;' '~
iw g .
- Q) ,
r,. .q#d , $ ( *
- f. ,
' ,8 1 *
}
a FCV 457 MOV 20 ) Figure 6.33 Inspection of FW Piping 5Y .. -. f l W ,
' c.4 '1 4
MINOR HANGER p' 2, i DAM AGE VISIBLE , , 7 . y , . , .
~
t ,l/ i 'l a / N,
.; E i $ i Figure 6.34 2-inch Bleed Line at FW Loop "B" k
i i H+. . , , . . _ , .l i
?hU$ . '
".;. \
l ,_ l
?c . -
l l . , FWS 376 l FWS 378 I (CHECKVALVE)
) ! FWS 342 . )
l l FWS 346 (CHECKVALVE) l l l
'i 3 5 l 1 ' N. ,. g l Figure 6.35 Overview of Valves FWS-342 FWS-346, FWS-376 and FWS 378 in FW Loop "B" Control Station l
I
2 l
~
- :,;. n
., 4 a - ..- - ' .i ,
~
, i.
- u* . e.,, },'
w i
** :% t 3 l m& - .!. , ^
,% 6 .v e
. J.R y{ ..
, - ~ .
. - ag, .
l I ,*p il 4 M/ d.T
.< ::5 q.. -
*d .
gNT .
- ~
. ^
; , g j s
! i
.'.t i.;
s t
- .s .
- e. .,
J Figure 6.36 FWS-378 Failed Bonnet d I i 1 i I i l l { I m.* .74 ;' Q - , ' rA.j' j .. r NOTE EXTENDED
' '9 - BOLTS AND i : " ~
. G ASKET EXTRUDED AGAINST BOLTS I l BY WATER HAMMER l
~- LOADING ,
. . \ m. . . fy.
e - # i %. b l :,..f.j?T
'. ' )
~, y.. .-
.*r. J *..
i Figure 6.37 Closeup of FWS-378 Bonnet t l l I _--,r- - --m,-- _ , _ _ _ _ _
l g Y-I. 4 *
.* ;a v . *T,'
4 ,
, , .s l Sr, .,
Figure 6.38 Damage to FCV-457 Actuator Yoke l l lk $ ' j 4 _'- .? _ y_
$4,
- n. ..
. _ ';% L'L -
- n. . f -,
.:. . ^': .'_'". .
~
['; * ' - e
.[' .
a: 4
- kyyzW e'?g . ~ -
- ) .
{'
.._io i . y, . j k(.'i li_ .
Figure 6.39 Closeup of FCV-457 Yoke Damage l l 1
l l l CONTAINMENT HELL l % ' ~ "~'"" N!" Efn'r^JE%f \ 1 y l , i h . AFW PIPING TEE Figure 6.40 Junction of AFW & MFW at SONGS-1 (FW Loop "B") frv e NI
*l CONTAINMEN j ,
. - , SHELL
.; l
- P s ..
l , I l 4 i i Figure 6.41 AFW Piping Run & Routing to Main FW Line (Loop "B") i
5 T 'p. , ' , - '.
. ' ~..J .
. 4' -
...'i.'.'
-,. . ge,
,4.,
l b, . &. yp_p.$... ,$. .m
- ). -4. - - M
, :,.. .. . .,k, , .=, ,,- . % y. ,.3..
-f . .
.'" ,\f _ f_ ' 4'_ .
\ a
.;. p ..
Y.
- w. -
-*v w .
<4.
VP - } g.
,.4% ./: "..'. j., *fV'.- '
i .. ;, I
, ,, g. I.. ,... .N. '.f .y* l
+
. . + w* h i- g s
'?c' ~ , . . .. : *~
, (* . . ( ,,
-( .
l
, P*Nm . . ,3l .', . , ,..;*# -
i Figure 6.42 AFW Loop "B" Support 1 l '1 ww _WW-W
~
. s. s , % L 'ma,.
^
). . .-
. ?i. ._' . ,l'; ,1. 4 l*-
e'l % ~ ':...v,$#! ' ly +-
, '% . . b - ' li + . .
y.u ~; <, - ~- - .- '%, J y . ,* .\. .
,'./..
_%5 s
. z7,- .
Qr
\,
?' d - .i*- ' y ~ ' ,? , ~
y N a.' f , . f m - . - m- # s, _ - , $wq?p. . , .,..
, y Q @ p., . . '- ..l~ * .-
_ l. .. - -g. jy ; - . l' h E _, p. ' 7- '
. e.] Wheb$' i ' ~ S' ~ W - l. s %l . ' '
~
Figure 6.43 AFW Loop "B" Motion Due to Water Hammer Loads I
l l 1 Y ! r.) ,o s
~ , !
j g.-
. L T-
i ::
e
, ~--
V. .-
~
^
. .' : .D
-r. .s.,
j . cg S,.} -
*lij *>
}'y!g
'.<N t
Figure 6.44 AFW Loop "B Piping Displacement Due to Water Hammer Loads i , m d . l
..w ;
1
' l-;; I l
1 l I.
'N; ;. . :
1 .. l l ! Figure 6.45 AFW Loop "B" Piping Displacement Due to Water Hammer Loads i
i VALVE BONNET BONNET STUD f a d kkkkk ve/'W glg
-HINGE NUT fr PIN ANTI ROTATION NUB d /
l _K CLOSURE DISK
,l N VALVE B00Y Figure 6.46 Typical Swing Check Valve With Internal Hanger, as Assembled.
,- ~ ~ ,
1 -
/ J L ti i s
L. _t - . l l ,Y
/
ANTI ROTATION
,s VALVE FWS 346 AS ASSEMBLED O
I
~
i
.y WORN PIN HOLE IMPACTED PIN f .
WORN THREADS I ' 4/ S N 7 VALVE FWS 346 AS FOUND Figure 6.47 Check Valve FWS-346
.i
. s -
;-g'? ne g,}*!}.f$U . . . ~ %~" 'hl{, -cl;;g:;Q.
j ff';. .. . - w+ ;,'- t -9 , , , ..
- NOTE DAMAGE (i.e., CUTS) TO DISC
, SEATING SURFACE l
h o/ 1 Figure 6.48 FWS-346 Seat Ring Damage 1
'5hh5[k nP y /,
p l a
., Y -
i A ,
'/j
- f* %
,,fs JN trf*pi s^
as- %*%}IN*f gfg '
/ ) _:. x .S k,. A l h v-. .'f Figure 6.49 FWS 346 Disc Face
_.--~ - - ----
l I i al . (~ . , }-} g ~ . . . . ( . .
' ^
i
..' ' %f f--
~.,s.
'f -
. f.
,1 a
.p. ., - .-
%y _ ), \ ,\
4' .._O} q ; ;,s)"* - j .. , L,*, ; w w.- Fe '!
. je .
1, '
}
.' ~
, NOTE IMPACT . - . J. '
3\ ' #" '-
? ' . _, %
DAMAGE AND .... !
.. j-l -s '
7 4 i WORN THREADS.
:. .~M ' .4 e
.j' S i- [
-. +k
' '. ;R'y - .,
; NUT WAS MISSING
. -[ s' T -
T - 3 . v. ,
.- , - 4 8, _
\ r ::
.i. g < : '.
. g, .
l 0 l ANTI HOTATION DEVICES , ,
. .. -[e
%,,, .eM~'. ?
- l
%g .. e, ,-
I.tf WJ 'J'- WOHN HINGE i #. A _.$
^ N' '
~ Q4 W;!- ~,
a i. - PIN HOLE $.
- . .~ # , -
j(.jl[,'[k#.ly
't
. .4 4 g. , ,
+ .. $ g ,gfk j
., , . .. e*< u .. ,
+, .
. *sg# _
=
f
+
, ', , y "
g 3 .1* ,
.... .. - 4;. .
* ~ ,
~
_4 h ' ' f_ ~ l _ _ _ __ _______;__[ l _? 'N :
~ ' V
~
Figure 6.50 Damage to FWS-346 Disc Figure 6.51 Worn and Elongated FWS-346 Nut Pin Hinge Pin Hole
4
.,Q f ,, f ?l.
. j q) ,- _
~
e .t'
). --
.s 4
-4
,s ~~,;^ .,'
.~
,, y.;,
11 4 .
- w. p
~
K g , 4 - ' *.- - J NOTE DAMAGED
,l 9 ' DISC STOP
. q .
.. ATTRIBUTED TO g
4 2. , ~ CONTINUED ! .. ,j " . ,e .
..: IMPACT BY
. '. CHECK DISC
/
j g ) %y - l
- s. x ,,
s
~
.c ,/) y ,
~'
.'l ..
- r. .r
'q o4.*.
- o g. ..
'i (4 - ,^
- O ., , @
s ,- m . . A M, p ~!,~1l.
'$..', E T' Figure 6.52 FWS-346 Bonnt.t l
l l
?
'.g
'*~
-- ,,. .y #
4~ ' y
=
7 x.;..
' ^
~
l Cl Figure 6.53 Closeup of FWS-346 Disc Stop
-w -,--m-,-m--- w- w - -w-a,,w. _ - - - - -w-_wm
.I
\
\
/
\
s
's s
n,,w., s. s .-
\
}(*
- 8 " ?{, . *
,\
/;;P k .
1
\ %. ~ \
1 4 7 1 i ' W "O 'k, * > 3 se 4$ 1 \
% o ~
r$ \\ eG Z d% === 5 \ ob or - b 5 e \ 5
.9
+
\ \
' o7
, v-f .
C r5 M d P0 G '" j 0 4 W 4
# */
d w g7O
i i 4
$' .,- ~ r .. ; u~ ~ . . -
p# .-(
~
. ]~
j
;;c 1
[ .- ,
\ . g '
j, ., 1 . M . r3; ': . i l '
',_ l% '[i.'
l 4 -
; f, s
i
., y. .
*j..
, k., 31ti !
- y. .;:
4 . . l, g
\' . <; , - %
.
- E.l. 'is >
. C. . ,
l 1 Figure 6.55 FWS-378 Check Valve internals I l 4 a,,_ .m.". . I
, #Wiiii.i
*9:1. . -
A~ .. '
*~' * .,
% - e I , _....-
l NOTE RADIAL R ESTR AINT PROVIDED TO GASKET By i BONNET GOLTS I L t
- I.
s .- J Figure 6.56 FWS 378 Gasket Following Water Hammer
4 I ! .,. ~ l i
- . ...- . 1
- ..- V* f. s s j l'g?
-q',4:.i.x f.
m, k ,,j .4
.;, - s.. .T
- s Wrv.. " ~# ej .ly.
, s.
_- c h. _k,
^ ~ , hi
.. . NOTE SEVERAL
'# H AIRLINE '
CRACKS ON l
, - w .#, '* ;'
SEALING FACE I T -
. Mi A
". l,,,, ,- . ( W ** l
.g. . , f.. . $
.c .,,.i c'>ef',, 's .,s..
Figure 6.57 FWS 378 Disc Face ' l l
/
; 'I f , --
;z ,
- a w- - ."' -
et NOTE ELONGATED l S '
.% ' ' . THREAD SECTION i
. % AT UPPER END l
+ .. . # ..- OF STUD
' - s
, h -
i . . Figure 6.58 FWS 378 Stud and Nut Removed From Bonnet 1 i o
4 .
,- m,
=
O ' I - i s Ny I i i i .
~
l l i t I \
~% ,
i VALVE FWS 438 AS ASSEMBLED l
,- m, O [ ,
~
w ' STUB \ CAUGHT \ I (,', UNDER HINGE 1 RO ATED FOUND '
~~
OPEN %
-15" j
\
1
, VALVE FWS 438 AS FOUND 4
Figure 6.59 Feedwater Swing Check Valve FWS 438
l t i c . 4 g* .. e y'
, ,, ./ - .i - .
r, .4 . , g, I i ..,,,. s .. ..-, _ 1
~. .m*d ' s, . '
. 3 =.
l ,.
.s .
~
l
!s ..
s ( , a fl b Q3; -. L* .
+
l f j , E, ;. . A. r, + 2
- s. .- , .
- 3. ; q s ' - , .
j i$ . ' ..#f 4 - -
,. / - 4 NOTE
/' t,g -y- ' ;fp g . A ,' ~
, . . # > ._ ? - F L A TT E N E D
, hi fg ' .: . .,,, 4 '; POSITIONER
?
ex. 1.i. 4 I-y ~
~;
e ~f
.. NUB
<': +;W h *, .' e. , . . . .. ::
f. c
' .vy. -% .k
...h
(
.+ -
l '
- .g.g; ;,qc ".. : ' ' ~
~ -
y .: '}q') r.; '- ~'--
=
y y,.'
& y'I.'! y ? .-
~ . ~ ~
\a ,
7 .- , _i
( ._ , j Figure 6 60 FWS-438 Closure Disc 1 l
- e. p. .
m- 9 s
,. $ m.eg -- . ..i.
- .* . e . v.
r
~
. i ,,
g .4 y .
- y
' . ,.5- ,
[k. Y . :.., [' Q.
,.. .. .P..e s a y.;,. y . . A 97, -
y. 1
>.. t s).*.* ,9 '.,2... .'
;% .,. L J, %.
a,
,r. , - ,
,c.. . , -
^
.'- J ~','..
# y _
4.,. f 6,
. , .g-4- ,
- o. .-.,. %.7 . ..g. y .
a .:: .. .
, *+*. -.
(
' ,' $g ; l , q ':: ; ,i l .
; $, ,.i sg . . . , .
? ,- N. . ,[ 3.. . *
,( .P .,
jy ,-
. . . . . . . .k. . .
- p , . 7 . ,., - -
. .. y :
.o pib .. ,y_ . , . ~,J,
. igure 6 61 FWS 438 Disc Holding Nut Arrangement
7 PERSONNEL PERFORMANCE AND HUMAN FACTORS EVALUATIONS 7.1 Introduction This section assesses the response to this event by utility and NRC personnel and the human factors issues affecting their performance. Other organizations, such as local, State, and other Federal authorities, were notified of the event, but did not play a significant role, so their participation is not addressed.
- 7. 2 SONGS Personnel Performance The utility personnel that responded included the on-shift operating crew, extra operators who were called in, and security and emergency services personnel.
Their performance and the human factors affecting their performance are dis-cussed in the general order of their involvement with the event. The Team interviewed most personnel involved and based the following assessment primarily on transcripts of those interviews. 7.2.1 Operator Performance The utility's immediate response to this event was made by the on-shift operat-ing crew. The crew was larger than normal because a Control Operator trainee and an extra NPE0 were also on duty. 7.2.1.1 Ground Isolation Actions Upon receipt of the ground alarm, the control room operators first identified the source of the alarm, used the alarm procedure to identify the appropriate response procedure, and then followed ground isolation procedure 501-9-7, "4160V and 480V Bus and Feeder Faults." Extra personnel, including one SRO-licensed member of management, and specialists in troubleshooting and repairing grounds were called to assist the operating crew during this phase. The ground isolation process proceeded as directed in the procedure through the minor loads until operators reached the <teps to test large loads, which verbatim are as follows: 6.2.5 REDUCE Unit load as necessary and DE-ENERGIZE the major equip-ment such as Circulating Water Pumps, Feedwater Pumps, etc. One at a time and observe the ground meter. NOTE: When a grounded circuit has been identified, it should be left de-energized until the problem has been identified and repaired. . I 6.2.6 When the grounded circuit is identified, then initiate a 1 Maintenance Order to have the circuit inspected and tested. ' 7-1
6.2.7 E the ground is on 1C or 2C Bus and cannot be located by de-energizing feeder circuits, then CHECK and RENEW, if necessary, the Ground Detector Potential Transformer fuses because a blown fuse will cause a false ground indication. 6.2.8 M the ground is apparently on the Bus, then with the concurrence of the Shift Superintendent, REOUCE Unit load as necessary, TRANSFER the Bus load to other Buses and DE-ENERGIZE the 4160V Bus. CAUTION If 4160 Volt Bus 1C or 2C is Inoperable in Modes 1-4, - then restore the Inoperable Bus within 8 hours or be in Cold Shutdown within the next 36 hours. (Tech. Spec. 3.6). - 6.2.9 Initiate a Maintenance Order to have the Bus inspected and the source of the ground eliminated. The actions operators took are identified in the narrative of the event, Section 3. The procedure provided clear direction to the operating staff. Based on inter-views, the Team determined that management permission to reduce power was granted, power was reduced and equipment was realigned to allow the testing of major equipment. Subsequently, all major equipment, except the feedwater pump, was tested and the grounded circuit remained unidentified. With the exception of some electrical instrumentation, the remaining potential locations of the grounded circuit were safety-related equipment, the deenergization of which would place the plant in a Technical Specification Action Statement, requiring that the equipment be restored to an operable status in a short period of time, or that the plant be shut down. At this point, the operating staff decided to deviate from the procedure, de-spite the fact that there was no valid immediate need to do so. A plan of action was developed and discussed with plant management, who approved its im-piementation. As a result, the following steps outside the procedure occurred:
- 1. Operators checked the ground detector potential transformer fuses before the procedure required that they be checked.
- 2. Bus 1C was paralleled with bus 1A momentarily to use an alternate ground indication to verify the bus 1C ground.
The period of parallel operation of auxiliary transformers A and C was
- short, but inappropriate, since auxililry transformer A already had an installed ground path, i.e., a second ground existed which could cause accelerated degradation of the existing problem.
Based on interviews with those involved, the Team determined that the operating staff became aware early in the morning of the event that the plant would prob-ably have to be shut down. The ground looked as though it was either on the feedwater pump or bus 1C itself. All required permissions, equipment alignments, power reductions, etc., were completed, satisfying the prerequisite to secure the feedwater pump. Management had been and was consulted again on how to deal 7-2
1 l with the ground. The interpretation of the Technical Specifications was dis- ! cussed and those involved recognized that the deenergization of the feedwater pump, bus IC, or auxiliary transformer C, would probably require the plant to be shut down. ' Preparations were made to shut down the plant when, based on further discussions, i the operating staff again decided to implement an ad hoc process to deal with the ground. This decision was made with the understanding that the planned ! activities were not described in the procedures and that the steps were intended to locate any ground that could be isolated without affecting the operating status
, of the plant.
As a result, the following steps outside the procedure occurred:
- 1. Bus 1C was again paralleled to Bus 1A and the PT disconnected to test the t' PT as a possible location of the ground.
This action again paralleled auxiliary transformers A and C, this time for about two minutes, and was inappropriate, as indicated above.
- 2. With the buses paralleled and the ground location still not identified, the tie breaker between bus 1C and auxiliary transformer C was opened; as !
a result, the indicated ground on bus 1C disappeared.
- 3. With the ground now known to be on auxiliary transformer C, the tie breaker was again shut to support further attempts to localize the ground.
This action again paralleled auxiliary transformers A and C, this time for about five minutes, and was also inappropriate, as indicated above.
- 4. Subsequently, the tie breaker between bus 1C and auxiliary transformer C was reopened, but the transformer was left energized while technicians went !
out into the plant in a further attempt to find the ground. The ground still existed on an energized circuit, sustaining the vulnera-J bility of the plant while inspections were performed that did not require i the transformer to remain energized. Further, inspecting grounded ener-gized circuits involves potentially significant personnel hazards. Deener- , gizing auxiliary transformer C would take the plant into an Action State- l ment and, as discussed above, might lead to a plant shutdown. ! i Following identification of the ground on auxiliary transformer C, the trans- f former was left energized and Inspections were conducted to further localize ; the ground. If the ground location had been identified and found isolable from l the transformer, the plant would not have had to be shut down. Both the failure to follow procedures and the actions taken that were not ' specified in the procedures were considered reasonable by the operating staff. Evidently, their training, plant knowledge and procedures did not alert them to the dangers of tying the grounded IC bus with the normally grounded 1A bus or working around grounded equipment. ! In addition, Procedure 501-9-7 does not provide instructions for the methods and steps to identify ground faults on a transformer. After the operators 1 7-3 t _ _ - _ - - - - - - _ _ - - - - . - - - - - - - - - _ - - . - - _ - - - - - - - _
localized the ground to the C transformer by steps outside the procedure, they had no training or procedural guidance on the significance of the ground loca-tion or the urgency with which the transformer should be de-energized. As background, Technical Specification Action Statements can be entered inten-tionally or because of malfunctions, and normally allow time for operators to resolve problems before plant shutdown is required. Valid purposes for occa-sional voluntary entry into Action Statements include preventive maintenance, troubleshooting, and surveillance testing. Based on the above discussion, it appears that the operators inappropriately
- deviated from procedures to unnecessarily delay entry into a Technical Speciff-cation Action Statement. Had the procedures been followed, the ground would have appeared to be on bus IC, the bus would have been de-energized, and the '
plant would subsequently have been shut down. 7.2.1.2 Recognition of the Loss of Vital Bus 4 and Reactor Trip When the initial alarms were received with the transformer trip, the operators assumed, based on their experience and training, that they were attributable to the loss of a vital bus. This was confirmed when they checked the vital bus indications on the electrical auxiliaries panel (Figure 7.4) and found that one of the vital bus availability lights was out. The operators needed to quickly identify which bus was lost in order to determine appropriate actions because the loss of some buses requires an immediate reactor trip while the loss of others does not. Efforts to quickly determine which bus was not available were ' hampered by labels which were too small to be read from where operators were positioned. The operators had to either go to the panel to read them or count the indicator lights on the panel to determine which but was lost. (Note: the I f ghts are in an unusual order--1, 2, 3, 3a, 4, utility, 5, 6. ) However, the operators were able to quickly identify the bus lost, and tripped the reactor about 20 seconds after the loss of the vital bus 4. 7.2.1.3 SI Verification The next operator action of concern was their verification of safety injection (SI) actuation. The second step in procedure 501-1.0-60, " Loss of all AC Power," is to verify SI actuation. The procedure specifies that the method of verifying SI is to refer to the SI Actuation alarm (RPF0-2) on the first-out panel. When the procedure was read aloud, the operators' first response was that no SI had . occurred, even though RPFO-2 was alarming. This SI actuation alarm was known by some operators to be unreliable because spurious alarms had been experienced with a previous loss of some buses. As a result, the alarm was initially assumed , to be spurious. However, it was then discovered that the SLSS remote surveil-Iance panel also indicated an SI signal. This panel (Figure 7.2) indicated SI actuation had occurred by indicating starting and loading of the diesel gener-ators, a condition that only occurs with a loss of power and coincident SI. This indication, together with RPFO-2 alarm, confused the operators and caused them to check the SI system operation. It was determined that SI was neither in progress nor necessary under existing plant conditions. While the procedure specifies one method of verifying $1 actuations, it does not give operators guidance on alternate methods to evaluate whether SI occurred, nor is there a caution that spurious SI indicators will occur on loss of certain 7-4
electrical buses. The operators' duties and responsibilities instructions in-clude guidance not to take actions based solely on a single indication but to use other indications to confirm readings. Although the operators did use other indicators, the procedures provided no specific guidance in this regard. During interviews, some operators also demonstrated a lack of detailed knowledge concerning the operation of the SLSS. 7.2.1.4 Electrical Power Recovery The next difficulty the operators experienced was with the station loss of volt-age auto-transfer scheme. The system has three primary indications associated with its operation (Figure 7.3). The first was received normally, indicating that the 18kV system had been isolated. The next expected indication was the - "open" light for the generator's motor-operated disconnect (M00) (Figure 7.4), which was also received. The last indication, before operators would normally take action to recover power, was the end of sequence indicator. This last light was not received as expected by the operators. After waiting for what the operators thought was enough time, they considered the sequencer faulty and took manual actions. Contrary to the procedures, the buses were re-energized bus by bus rather than all buses being tied together before th' system was ener-gized. The operators were unsuccessful in three attempts to return power to the site using the 220kV circuit breaker (CB) 4012; the first attempt with the 220kV C8 6012 was also unsuccessful. The second attempt with C8 6012 was suc-cessful and the buses were then energized. C8 4012 was closed later. The investigation to date has not determined whether or not the system operated as designed. However, no failures of this system have been found. It appears, at this time, that the operators did not know the time required for the auto-matic system to operate. The decision to manually re-energize the buses rather than follow the action described in the procedure seems reasonable, but it re-suited in errors caused by the lack of detailed information that was implicit in the procedure. The attempts to close breakers C8 4012 and C8 6012 failed for two reasons. First, the initial attempt to close both breakers was not performed properly. Before energizing one of the dead buses, the interlock checking the synchroniza-tion across the breaker had to be overridden. The operator did not use the bypass pushbutton for this purpose on his initial attempt to close each breaker. The 4kV breakers normally manipulated by operators do not contain this inter-lock feature so the operator was not familiar enough with this unusual operation to remember the bypass without being reminded once during each attem)t. The same operator later used the bypass to close C8 4012, even though both sides of the breaker were energized and the interlock should not have been defeated. The second attempt at closing CB 4012 failed because the initial electrical-trip signal had apparently not been reset properly. For the lockup bus to be reset, the turbine generator motor-operated disconnect (M00) had to have completed its opening. Since the operator attempted the reset early in the event, and because the MOD opens relatively slowly, the first attempts to reset the lockup bus were probably not successful. Further, the operator did not verify that the resets were successful by either observing the light around the reset button itself or by observing the clearing of the alarm on the vertical panel. The procedure, had it been followed, would direct operators to reset the lockup bus after the MOD had opened. Because the procedure was not followed in the order specified, the initial attempts to reset the lockup bus were not successful. 7-5
These operator actions demonstrate thei* lack of knowledge of the operation of the synchronization bypass switch, the lockup bus reset circuit, and the station loss of voltage automatic transfer scheme. 7.2.1.5 Use of the Diesel Generators Using the diesel generators to recover electrical power was considered by the operators during the loss of inplant power. Because operator training and the procedure for the use of the diesels (501-1,0-60) gives priority to available offsite power as the preferred source, the operators continued their efforts to re-energize the buses from the switchyard. Because inplant power was lost for , approximately 4 minutes, the delay in re energizing the buses raises the ques-tion as to how long the operators would have continued in their attempts before using the available diesel generators. The procedure, however, provides no , i guidance on how long the attempt to restore power from offsite power sources should continue. The Shift Superintendent told the Team that he was aware of , l the difficulties of restoring power from offsite and would not have waited much longer before acting to load the diesels. The procedure utilized by the operators did not specify the maximum period of diesel generator operation while unloaded, with or without ac powered auxil-laries. The radiator fan motors must be powered in about 39 minutes to avoid diesel overheating, and long-term diesel operation at no load is known to be deleterious to diesel operability. The previous version of Procedure 501-1.71, Rev. 2 (Dec. 1981), had the necessary guidance, but it was dropped when the procedure was revised. 7.2.1.6 Initial Response to Indications Af ter the reactor and turbine trips, the behavior of the primary and secondary systems was evaluated based on the operators' experience and training. The review of major plant parameters revealed to the operators that the pressurizer level and the steam generator levels seemed lower than expected, although both were not unreasonably low given the operators' understanding of the plant status. The low pressurizer level was attributable to a rapid cooldown, although it was also recognized that charging had stopped with the loss of power. The concern about the lower-than-expected steam generator levels was offsot by the realiza-tion of the design delay in initiating auxiliary feed flow from the steam-driven auxiliary feedwater pump and that the motor-driven auxiliary feedwater pump was off during the loss of power. Operators dealt with pressurizer level first by manually starting a charging pump. The second pump automatically started by design in response to the low . discharge header pressure. The trip of pressurizer heaters on low level (less than 10 percent) increased operator concern for level and pressure control and raised the possibility of an SI initiation. The decreasing pressurizer level was evaluated as being caused by the cooling ' of the RCS by the steam generators. Accordingly, the control operator respon-sible for the RCS directed the operator at the auxiliary fledwater controls to ,
" cut off" auxiliary feedwater (AFW) flow to avoid an SI. The concern was based l on the decreasing pressurizer level and knowledge tnat a low pressurizer level l would lead to a further decrease in RCS pressure, possibly to the SI actuation l setpoint. Subsequently, AFW flow was reduced from approximately 135 gpm (indi- <
cated) to zero for about 10 seconds before the Shif t Superintendent directed ) 7-6 1
that AFW flow be re-established at 25 gpm, because of his concern that the steam I generators would become dry. (Note: steam generator level was already low and ' decreasing.) This indicated rate of flow (actual rate was determined later to be approximately 40 gpm) met the procedural criteria to avoid declaring a steam generator " dry". A " dry" steam generator is defined in E0I S01-1.3-3, " Response to Steam Generator Low Level," a: not having any of the following:
- 1. Wide range above zero inches.
- 2. RCS loop differential temperature greater than zero.
- 3. Feed flow equal to or greater than than 25 gpm.
These actions demonstrated a reasonable concern for the RCS status over the steam generator status. The interaction between the operator responsible for
- the RCS and the operator at the auxiliary feedwater controls is expected and the Shif t Superintendent's actions demonstrated his " big picture" view of plant operations.
7.2.1.7 Decision Not To Recover Normal Steam Generator Water Level After pressurizer level was recovered, reactor coolant pumps A and C were started, and with steam generator water level still observable, the plant was in a relatively stable condition with the cooldown continuing. The procedural guidance for the low steam generator water level (less than 10 percent on narrow range) is to raise level to 50 percent on the narrow range (50I-1.3-3). The procedure contains no cautions concerning the effect of this level change on the reactor coolant system. Several operators, the Shift Superintendent and the STA discussed the inadvisability of this action and what direction to take. There was a known steam leak in the mezzanine area but not specific information I as to whether the leak was in the steam or feedwater systems or which loop was affected. However, operators observed t hat although steam pressure was low, the leak was not large enough to depreshirize the steam system and was not caus-ing an excessive cooldown. (A steam generator pressure less than 400 psig is the point at which operators refer to procedures for loss of secondary coolant.) The operators decided that a cooldown could be conducted safely, with a com-fortable margin to the 100-degrees per-hour limit in the Technical Specifications. This cooldown was initiated by increasing the feed flow to the A and C steam generators since they had their reactor coolant pumps running. This process continued until the plant was placed on RHR. 7.2.1.8 RHR Initiation Operating Instruction 501-4-9, " Residual Heat Removal System Operation," con-tains a prerequisite (3.6), and a step (6.1.8) to close RHR-004, the hot leg recirculation bypass valve around the east RHR pump. The valve is normally open to allow a flow path around the RHR pumps to support the post accident hot leg recirculation use. The relatively routine RHR use can be accommodated by shutting the valve, which must be locally operated within the containment build-ing. The operators considered that the plant was well enough under control that the procedure should be followed. RHR allonment encountered difficulties because of a pressure interlock. To protect the relatively low pressure RHR system from the high RCS pressures, one of the RHR inlet isolation valves, MOV-813, and one of the RHR outlet isolation valves, MOV-834, have an opening interlock based on pressurizer pressure. When 7-7
r the operator attempted to open MOV-813 per step 6.1.6 of 501-4-9, it would not open. The operators considered entering the containment building to operate the valve, but dropped the idea when they found that the interlock relay was readily accessible. The RCS pressure indicated in the control room was 370 psig. Since this pressure was below the operators' understood relay setpoint of ! 400 psig, the relay was locally operated using a contact follower button to allow the opening of MOV-813. The relay also prevented MOV-834 from opening and was again overridden. Post-event investigation found that the relay was aligned properly and would not reset until pressure dropped to approximately 367 psig (see section 8.8). ' The reason operators believed the set point was 400 psig was attributable to their procedures and training. The procedural reference to 400 psig is in-cluded in step 6.20 of 501-3-5, " Plant Shutdown from Hot Standby to Cold Shut- - down," in 501-4-9 as prerequisite 3.5, and the note following step 6.1.3. It is also included in training study guide number 22. Therefore, the operators had no reason not to expect the relay to a110w the valve to operate under exist-ing plant conditions. 7.2.1.9 Continuation of Feed Flow to B Feedwater Line The operators' joint decision to continue feeding the 8 feedwater line after a leak was identified was based on concerns for the safety of personnel attempt-ing to locate and isolate the leak. Their intent was to cool the line feeding the break and thereby change the escaping fluid from steam to water. However, unknown to the operators, this action could have created conditions for a second water hammer (from the continued injection of cold water into a hori:ontal feed line with steam voids). Although there is procedural guidance on a maximum feed-water flow rate into a steam generator with a low water level to avoid stean generator water hammer, there is no guidance and apparently no training for operators on an acceptable flow rate into a voided feedwater line to prevent feedline water hammer. 7.2.2 Other Site Personnel Performance This section addresses the performance of site personnel other than operators who participated in the event. They include the STA, emergency coordinator, emergency services, and security personnel. This section also addresses the post-trip review. 7.2.2.1 Shift Technical Advisor's Performance The STA participated significantly in this event and took part in major deci- , sions made by the control room operators. After assisting in the search for the ground fault, the STA informed the Shift Superintendent that he was going to bed. He had gone to sleep in his trailer at about 04:30, when he was awak-ened by a loud noise. After attempting to reach the control room by phone twice unsuccessfully, he started dressing. He also inspected Unit 1 from the STA trailer and observed nothing unusual. He then heard on the two-way radio an announcement that a fire truck was headed to Unit 1, and started for the control room in earnest, arriving at about 04:58, according to the control room clock. Since that clock had been de-energized for about 4 minutes during the loss of power, the STA arrived 11 minutes af ter initiation of the event. The STA is supposed to arrive in the control room within 10 minutes of such an event. l 7-8
He exceeded that requirement at San Onofre because he had not been alerted that the event had occurred. On arrival, the STA began performing his normal post-trip task of checking the
" Critical Safety Function Status Trees," S01-1.0-1. His only concern was that the steam generators were below the 10 percent water level on their narrow range instruments. Since that condition was common following reactor trips, the STA ,
reported it to the Shift Superintendent and they both agreed to look into it I later. From that point, the STA made periodic independent checks of conditions in progress, referred to the Critical Safety Function Status Trees, participated in control room decisions, and checked the guidance of other procedures. The STA contributed to the decision to cooldown at less than the maximum possible rate, specifically to stay well within the 100 'F per hour limit. A rapid cool-down plan was considered, along with recovering steam generator level, after the report of the steam leak in the mezzanine. But, because the plant was relatively stable even with the reported leak, the STA recommended that rather than going through another transient that the plant continue to cooldown within the 100 *F per hour limit. Once this alternate plan was adopted, the cooldown was controlled with auxiliary feedwater flow. He checked the procedure for low steam generator level to see if there was anything else they should do to re-cover level and found the reminder to secure steam generator blowdown. The l operators then recognized that the steam generator blowdown at about 100 gpm i
; for each steam generator had been inadvertently re-established shortly after the restoration of electric power, and immediately isolated the blowdown by reducing the radiation monitor setpoint. (The status of steam generator blow-down is not indicated in the control room.)
By all reports the STA performed the useful function of independently monitor-ing operating crew activities and made a positive contribution to the shift's performance. 7.2.2.2 Emergency Coordinator
! The Emergency Coordinator's responsibilities are normally fulfilled by the Shift Superintendent until he is relieved by an operation's management representative.
The acting Unit Superintendent, an individual normally outside the group con- 1 sidered operation's line managers, was on site to assist the Shift Superinten- : ! dent in troubleshooting the electrical ground on the IC 4kV bus. This individ- , i ual returned to the control room during the 4-minute power loss and, after power I was recovered and the situation was evaluated as an Unusual Event, took over the position of Emergency Coordinator.
- NRC was never officially notified of the Unusual Event declaration, a responsi- '
bility of the Emergency Coordinator, although several phone discussions were held between the site and the NRC. The likely cause of this oversight is that the official declaration to the NRC was simply missed. The Emergency Coordinator provided the first full explanation of plant conditions to the NRC. The explanation included three major errors which contributed to NRC's confusion. First, the reactor trip was explained as resulting automati-l cally from the loss of the transformer. Second, the plant was confirmed to be ' in an Alert status and third, the feedline leak was initially described as a leak in the main steam system as opposed to a steam leak of unknown origin. 1 i 7-9 {
Thus, there were inadequacies in the manner in which NRC was notified and in how the event was described. 7.2.2.3 Other Support Services Other support services were provided by emergency service personnel, health physicists and security personnel. Emergency services personnel responded to the event when the possibility of a fire was reported and subsequently partici-pated in attempts to locate and secure the source of the steam leak. No difff-culties were noted in the performance of the emergency services personnel. Health Physics personnel took samples as requested. With the loss and restora-tion of power, security systems were affected. Site security personnel and operators recognized the situation and implemented planned compensatory measures. - No significant problems were encountered by plant staff in gaining required access to safety-related equipment, and the procedures appeared to work smoothly. No difficulties were noted in the performance of health physics and security personnel. 7.2.2.4 Post-Trip Review The unavailability of the critical function monitoring system provided by the Foxboro III computer impeded the post-trip review process due to the lack of data to accurately reconstruct the event. The operators' failure to have the computer reset after its power supply was interrupted during the troubleshooting activities and af ter the 4-minute loss of power negated the data gathering and recording functions of the system. Consequently,,the trend recorders provided the only plant thermal-hydraulic parameter data available for the evaluation of the system responses. As the name implies, these recorders provide only qualf-tative information. During the 4-minute loss of ac power, the computer and , most trend recorders did not record useful data. In discussions with SCE technical representatives, the Team observed that, on occasion, some site personnel who generally evaluate plant data lacked a suffi-ciently inquiring attitude. As a result, certain significant underlying reasons for system response or component performance were not detected until brought to SCE's attention by the Team. Examples include (1) the Team's reluctance to accept that the flash evaporator failure was not in some way connected to the water hammer led the Team to hypothesize a connection, which when tested, led to the discovery of the first failed feedwater pump discharge check valve; (2) the Team's reluctance to accept the first explanation of the similarities of east and west feedwater pump suction temperature traces led to the discovery of the second failed feedwater pump discharge check valve; and (3) the Team's reluctance to accept operator log entries which could not be confirmed on every
- pertinent recorder trace led to the confirmation of the Team's conclusion that a reactor coolant pump had been started earlier than reported. It appears that SCE's process for evaluating and following up events may not be suf ficiently thorough, and not systematic enough, to identify all failed components and root causes of failures.
The SCE's post-trip review report has not been completed. (The unit will be in a scheduled refueling outage for several months.) 7-10 i
- 7. 3 NRC Emergency Response Performance NRC involvement in the November 21, 1985, event was limited to receiving the initial notification of the plant transient, alerting appropriate staff members, establishing communication Ifnks, dispatching resident inspectors to the site, asking questions, evaluating information, notifying other Federal agencies, monitoring SCE activities, and answering questions.
7.3.1 ENS Communications Problems , The communications problems between San Onofre Unit 1 and the NRC can be divided into malfunctions of ENS system hardware and poor communications on the part of site and NRC personnel. The hardware problems with the ENS are discussed
, in section 8.10 and the site's communications performance deficiencies are dis-cussed in section 7.2.2.2. This section addresses the NRC's part in the com-munications problems.
Several of the reasons for poor communications on the ENS are NRC's responsibil-ity. The NRC pursued a Ifne of questions that focused on details rather than on the overall plant status. The information provided to NRC frequently was not understood because the recipients lacked site-specific background information and were therefore confused about plant conditions. The NRC asked leading ques-tions that produced answers of If ttle value or created misconceptions about the sequence of events. The NRC did not establish an open phone Ifne to the site for over an hour. NRC Resident Inspectors took over the communications function from SCE personnel and then transferred the phone pickup location to a point remote from the information available in the control room. Finally, the NRC used the ENS phone for purposes other than obtaining information about the plant, f.e. , a redundant retelling of information for each new NRC participant who came on the line in the ongoing discussion. 7.3.1.1 The Focus of NRC Inquiries In reviewing the transcript of ENS communications, it became obvious to the Team that questions asked by NRC characteristically focused on detail, when frequently a better question would have provided an overall perspective of the plant's status and the sequence of events which produced it. For example, NRC learned that the plant had experienced a loss of power and that the reactor had been manually tripped. Instead of asking why the plant had to be manually tripped, NRC asked whether they lost flow. The affirmative response by SCE led NRC to conclude that the reactor protection system (RPS) N st have failed, because a loss of flow should produce an automatic trip. In reality, a partial loss of power had led the operators to manually trip the plant, which led to a loss of power to reactor coolant pumps, which in turn led to the loss of flow. Nothing was wrong with the RPS, NRC's information gathering approach arises from the common practice of attempt-ing to understand events in a causal form. This event-based reasoning usually works well in an engineering environment, where engineers evaluate the effects of component or system failures, f.e., when reasoning from an event forward. The approach falls when the task involves reasoning Dackwards and the initial cause is incorrectly diagnosed or when thero is more than one significant cause. Because faulty reasoning was a major contributor to probloms in the response to P11
~
t e the TMI accident, NRC required the industry to develop another approach for operator control during emergencies. This approach has operator procedures focus on symptoms or safety-functions as the basis for emergency operator actions. This same concept may be of benefit to NRC staff involved in ENS communications during emergencies. 7.3.1.2 NRC Lack of Plant-Specific Knowledge NRC at times misunderstood statements by SCE because the recipients lacked know1-edge about the unique design and operation features at San Onofre Unit 1. For example, the plant does net automatically load diesels on a loss of power and, under these conditions, plant-speciffc procedures require use of off-site power, if It is avalaible. The NRC Duty Officer assumed that the diesel generators ; should automatically load on loss of power, as is common in the industry, and
- therefore assumed that the recovery involved loading the diesels. The site identified the specific transformer lost, but that information was not useful because the NRC did not know the plant-specific electric bus configuration or identifiers. When the steam leak was reported, the NRC asked which side of the main steam isolation valves the leak was on. This plant does not have main steam isolation valves and must manually shut block valves.
These and other examples of plant-specific knowledge deficiencies caused mis-understandings that contributed to NRC confusion on what happened. 7.3.1.3 Leading Questions NRC frequently asked leading questions designed to elicit responses that would support assumptions about how the plant was designed and operated, or hypothesis about the sequence of the event. Unfortunately, the SCE staff did not always catch the errors in logic, and therefore did not correct them; on occasion SCE appeared to confirm inaccurate information, as in the following excerpt: NRC: OK, Did diesels pick up? Site: The diesels started. NRC: Old they load? 5ito: No. They don't automatfcally load here. NRC: 0K. So that was part of the recovery process Site: Right. . NRC misinterpreted the response to the implied last question as confirming that the Inplant buses had been recovered by loading the diesel generators. The inplant buses were recovered using the prefstred source, the switchyard, as designed and required by procedure. Unlike most other reactor plants, these diesel generators start on loss of power, but do not automatically load unless a safety injection occurs. 7-12
Another example follows: NRC: Now, the basis for the Alert, that's because you had a loss of offsite power? Site: It was based on that, that's correct. NRC: You did lose of f-site power? l Site: We lost off-site power to the site for approximately 15 minutes. NRC: Was that only to Unit 1 or to the entire station? 4 Site: No. Only to Unit 1. NRC: Are you out of the Alert now? Site: We have not gotten out of the alert right now. We're still eval-uating. As I indicated to you we have a break in the main steam system. And we're evaluating whether we're in a UE [ Unusual Event] due to a rapid depressurization of the main steam system, i NRC: OK. So you're currently in an Alert. Site: Right. Later, after NRC informs FEMA that the plant was in an " Alert," the following conversation occurs. NRC: The Ifcensee has been ervergency borating, they have aux feed-water, is operating satisfactorily. The Ifcensee, correct me if I'm wrong, declared an Alert... Site: No sir, I declared an UE. NRC: An Unusual Event? Site: Unusual Event. NRC: Are you in an Unusual Event? Site: No Alert was declared.
. NRC: Not declared.
Site: We were never in are Alert. We are still in an UE. And we are probably close to getting out of it, now that we've isolated our steam leak. The combination of leading questions and confirmation of misinformation signif-icantly confused NRC's understanding of the event and plant status. In the - first instance, the site appeared to understand the term " alert" as a generic name for emergencies. This may be because the individual responding to NRC 4 questions does not normally fill the role of Emergency coordinator. However, Communication difficulties of this kind are attributable to the inadequate 7-13
training of ENS system users on the subject of how to use the question and answer process to obtain appropriate, timely information. 7.3.1.4 SCE's Inability to Support an Open ENS Line The NRC perceived statements by plant personnel to indicate that they could not support a continuous open ENS line. The first call occurred during the loss of ac power and the Shift Superintendent hurried the NRC off the line to get back to the plant. Near the end of the second call, there was a concern in the con-trol room for a possible SI and the Shift Superintendent again cut the call short. In the third call almost an hour after the trip, originated by the NRC - Duty Officer, the Duty Officer requested that an open line be maintained. When he discovered he was speaking with the Shift Superintendent, he suggested that the Superintendent call back when they could support the open line, even though - the Shif t Superintendent volunteered to provide a summary of plant status. The Superintendent then agreed to call back. In the next call, the site's Emergency Coordinator stated that he did not understand the event yet and asked if he could call back in 15 minutes. (However, the NRC continued to question him regarding the cause of the event and in response he provided information which later turned out to be inaccurate.) The NRC Headquarters Duty Officer and Emergency Officer took the first two calls to mean that the site did not have sufficient personnel to support an open ENS, which seemed to be confirmed by the subsequent calls. This conclusion resulted in limiting the initial communications with the site unnecessarily. 7.3.1.5 NRC Site Communicators Another communication difficulty resulted because NRC Resident Inspectors re-11eved more knowledgable plant operators as ENS communicators. This substitu-tion deprived the NRC staff on the ENS of access to personnel knowledgeable about the plant and capable of interpreting plant data. Further, NRC site per-sonnel used a telephone remote from the control room to communicate with NRC headquarters. (See Figure 4.17 for the location of the phone used in the NRC consultation room.) The NRC rationale for the use of this phone was to minimize crowding in the control room. As a result of this relocation, runners had to be used, as available, to take requests to the control room and return with the relevant data. This cumbersome arrangement resulted in the inability of NRC staff on the ENS to monitor plant conditions in a timely manner. Because SCE had staff and the responsibility to support an open ENS Ifne, it was
- unnecessary and inappropriate for NRC to assume the role of the lone communicator.
7.3.1.6 ENS Conferencing
- The ENS system was used by NRC for function: beyond initial notification and information collection from the site. As each new NRC participant was brought into the conversation, he was briefed about what had happened. During these discussions, little new information was revealed and the site communicator was frequently a passive participant. When briefings were not being conducted, the conversation frequently turned into a conference evaluation of past data, some-times to the exclusion of soliciting new information from the site. Both uses of the ENS would have been better handled on separate telephone Ifnes and doing so would have allowed the establishment of systematic processes for obtaining 7-14
information needed by the NRC to understand the event. The situation appears to arise because NRC lacks policy and guidance on the prcper use of the ENS. 7.3.2 NRC Incident Response Plan Implementation During the event, the NRC partially staffed the Region V and Headquarters Incident Response Centers (IRC) but did not formally enter a standby response mode, as described in NRC Manual Chapter 0502, "NRC Incident Response Program." Early communications between the site and NRC were not effective in developing an understanding of the event. The need for additional information and clari-fication of previously established facts was recognized by both the Region V Duty Officer (RDO) and Headquarters Emergency Officer (EO). The RDO informed
- the Senior Resident Inspector (SRI) of plant status and his concerns, and the SRI volunteered to go to the site to determine what was going on and then call back. The E0 requested that additional staff members go to the headquarters IRC to establish an open communications line with the site to further evaluate event information. Subsequently, an open line was established between the site, the Headquarters IRC and the Region V IRC. As the morning unfolded, additional NRC personnel staffed these three communications centers.
SCE indicated in the second call that an Alert emergency classification declara-tion would probably be made on a subsequent phone call, and that it would prob-ably be closed in the same call. During later phone discussions, SCE did not notify NRC of the declaration of an Unusual Event. At 05:06, and shortly after 06:00, SCE personnel appeared to confirm that the plant was still in an Alert. Based on this information, the HQD0 notified the Federal Emergency Management Agency of the event at San Onofre, Unit 1. Generally, a site declaration of an Alert would precipitate an NRC response mode transition to standby. This transition can be authorized by an executive team member, or Regional Administrator, or in the event of their unavailability, by the E0. Approximately 15 minutes after the apparent confirmation of an Alert, the Regional Administrator was bridged into the open line and, in response to his request for a summary of what had happened, learned that the site was in an Unusual Event. With this clarification of event classification, and recognition of the current status of NRC response, no formal activation of the incident response plan or transition to standby was warranted. The activities being performed by NRC were appropriate to a normal response mode. 7-15
l l 1 ! // I II I I I
~
l l w 2 b. l l Fig. 7.4 Fig. 7.3 Fig.7.2 h l l 1 - w I ! l - - Q l l Figure 7.1 Control Room Orientation of Figures 7.2,7.3, i j and 7.4 l i i ! I i i
*' e 4-
.:. {
>;,~ - , l.
f e :. p: - t s
)
j l,y'%. .,.R.~>.'*+_' ei),
- gg i i
SEQUENCER LIGHTS OUT INITI ALLY l I y: : l'. Figure 7.2 Safe 0uard Load Sequencing System (SLSS) j Surveillance Panel i. 1 4
CB 4012 q
- t. f <_ g '.. '. . - . ; ;. >
. .$ % ~;~# * .
BREAKER >.
.. c t*
l CONTROL i -
+ 4.-
i-
;).. fg. .f(. ;- p' . )'jlr ..y' ,*Q h '8th' ,%
AND
+ -
,6 f . .- j- .
lNDICATOR ?e 1 + t q. j! y.,: agi', : -r 'J. . :h' g j -
, ll 3. . . . [ . /
; . es. .. : , ' . .
" SYNC. [" ~ *k
- A , , , . .
- 7. -
i;. ' 8Y. PASS" . I . f(s. . , l ..' .g. l
.p,s a ,>. .s. .
.o -
+ s ,?. y .1L % l.l-p 4.. , L. * . s . .. .
.3 ,.
.. , '* AUX Y
' '$+ /* TRANSFORMER
- w. : p' . y CB 6012 f p' * .- -
BREAKER a : M4 e
. . . ,W .
C CONTROL AND
* ~;. _ * ' ' f%
- ;' ?
i.
~* i.~'
~
. . N{n ' '
g ;. l '.- . . . jy I INDICATOR :...
d >
," P.: - '
7 - l
.s[,
l $. ). l#. .'[.
*- . -- f c ,
~' I - . [. t T.):. r' - -3'
" LOSS OF 220K V
'...y,
" -, '
- _ . +: - ' p . ! li. '
+ AUTO l w
,$ . *; e ~ . , -
~
f l TRANSFER
- M' + f,,
,, ' z; . . . 1. ..
END OF
, (,+ - '
SEQUENCE"
- n;
.e ,
..... / , . . . . .. . _ - . _ . . . ..
" LOSS OF 220KV 18KV SYSTEM Figure 7.3 Electrical Auxiliaries Panel (left side) ISOLATED" c
l , i l VITAL BUS AVAILABILITY LIGHTS (1. 2. 3. 3A. 4. UTILITY AND 5, 6)
. w WHillB - W I
1 MOD OPEN INDICATOR gg 9 cs AUX. - ~ . TR ANSFORMER I8 . Y ff!> 1 w l l l l l Figure 7.4 Electrical Auxiliaries Panel (right side) 1 1
1 i fyq; l > S' ,;~ ~ 4- -
,, . r, o w..-* e.%,
, g N. s.
% ' f. g
/ .- %
. ur ..,
h*.&s
,p M '
i Figure 8.1 Cable Length Showing Damage 1
, ~ ~'; _.. .}'X :..':.,";l-
~
^
g
- - 1, .,
.. ^c
--[,_-.;q.; _
,- . } :; [;. :(( .
I' ~ [~l7.)[I.-[7 :[2;L,{
- 1
; _,+ ;:
; - ', fg 7.[;]: j .j .f : J. [ '
&_q: ff . .. s.y . . ,' .:. , .
J N. ( .. .. . . . .
'( - wci. ,,
- g ' . -- ,
9 l *z ,. . ^- ' , . , ,
~
c ., . -.
. g"w
~ _. _ ,
.. .', 1. <.a . ;
- f. - .- 1 J
...,c....
K. 4
,f ~ , ~ + &,
4'.
. . .y, ., , . ..%: i i m
.-4 4; '
g
.f ,;; s. . *
. , . ,. y
- 7. -
l ue -
.; 7 g , ' .-
. . +-
'*jf . . , , ) N . y l
Figure 8.2 End View Showing 3/C 750-Kc mil Copper Conductor, Neoprene Jacket and Aluminum Armor
l J r .
~r v
? 'l
- Y Y@
' ' ,W'i+h;;?h.Y'i&.,.j%yg,
;47,.~g- >h*&~
.. , ;i= '>% .'E f:".'h-J d[Slk:$$?
%;l3p.;m.m :
T:?.,%YUdN!f #' j$o hh 7 rg. < a 3. -
*t e~ ,ry;2 ' * .i f.i'?i% 4ft!e A-q,,,',
t I i !
\
l
;.,,p-
, g{, , 't, j,. [L,,~T .pa.1;. F.ewvs-L,w .
2 . ...! Figure 8.3 Cable Section When Phase-to-Phase Faults Occured
'h'N7,'g,[. . . -is . Q;)bk; R ; , ,
r& i , ~ m
'k,~.,
' M@c i.f y. 4. - ,
v,
$? $$
; . :,s. o fikw_.
Y
.'.4%FU2'.s~1f',i,.
*e. *r,--.
Y ~
\* ,.*
y , ?'mgW,;. '. .~ - e ,e
-, ?l5.WjN.1\b.h * '. '
ut4gqiy
, n.b,r.
- p. .. .
,, .;
- vs .. +WA .~9, s
-s, .
-h Figure 8.4 Close-Up of Damage Caused by the Fault l
\
8 OTHER EQUIPMENT AND SYSTEM EVALUATIONS This section identifies and evaluates other equipment and systems that had problems during the November 21, 1985 event. The performance of the equipment and the root cause of the problems are also included.
~
8.1 Auxiliary Transformer C Secondary Side In the process of searching for the ground fault on 4160-V bus IC, plant per-sonnel had located it on the X winding (secondary side) of auxiliary transfor-mer C between the transformer and the bus. At 03:35, the operators had isolated the fault from bus 1C, but kept the transformer with a ground energized. At 04:51, the auxiliary transforrrer C differential relay actuated, tripping and isolating the transformer. After the trip, the targets on the differential relay identified that phase B and C trip had occurred. Subsequent investigation has located the fault to be in one of the interconnecting cables between the transformer and bus 1C. The interconnecting cables are 3/c-750 Kc mil aluminum armored cables, which run in cable trays located in the turbine building. The cable tray which contained the faulted cable section was located directly be-neath a feedwater pipe flange, which appears to have leaked for sometime. Fig-ures 8.1, 8.2, 8.3 and 8.4 show the details of the damaged cable section. A detailed examination of the damaged cable will be performed ,by SCE. The root cause of the transformer differential protection actuation is believed to be the phase-to phase fault in the cable section which was damaged. The cause of the cable failure is under investigation; however, the intrusion of water into the cable could have contributed to its ultimate failure. 8.2 Safety Injection Annuciator During the event, when the unit was without ac power, the reactor plant first-out annunciator window 2 (RPF0 2) alarmed indicating SI initiation. The control room operators reviewed plant conditions and concluded that the alarm was spur-ious, and that SI had not initiated and was not required. Subsequent investiga-tion has determined that the alarm relay that actuates the SI annunciator window is ac power dependent and will erroneously alarm on loss of that power. This is a design deficiency.
- 8. 3 SLSS Remote Surveillance Panels During the event when control room operators were verifying SI actuation, the SLSS remote surveillance panel load group status lights were reviewed for indication of SI actuation. The load group status lights on both sequencer l
panels indicated SI actuation. (Section 4.13 describes the operation of the status lights.) This indication of the SI actuation was also determined to be i spurious, but did cause some confusion to the operators during the event. This spurious actuation of the SLSS surveillance panel lights is under investigation by SCE. 8-1 l
o 8.4 Flash Evaporator Unit During the event, the east condensate header was overpressurized causing a catastrophic failure of the east flash evaporator shell. As shown in Figures 8.5 and 8.6, the evaporator unit consists of a flash evapo-rator in a common housing with the 4th and 5th point low pressure feedwater heaters and drain coolers. The flash evaporators have not been used for several years and extraction steam to them has been isolated. The evaporator condenser is, however, still part of the condensate system flowpath. Design pressure of - the flash evaporator condenser, 4th and 5th point low pressure feedwater heater tubes is 350 psig, while the shell side design pressure is 15 psig. The low pressure feedwater heaters were in service on November 21.
- When bus 2C deenergized and the east main feedwater pump tripped, failed dis-charge check valve FWS 438 allowed the west main feedwater pump to pressurize the east condensate header. This pressure caused a tube failure in the east evaperator condenser which pressurized the flash evaporator shell resulting in the failure of the shell shown on Figures 8.7 and 8.8. The evaporator heater tubes are visible on Figure 8.8. After the loss of all inplant ac power, the remaining (west) main feedwater pump coasted down, and failed main feedwater regulating valve check valves (FWS 345, 346, and 398) allowed backflow from all steam generators through the failed east and west main feedwater pump discharge check valves (FWS 438 and 439) to the failed tube in the east flash evaporator condenser. This backflow continued until the operators closed motor operated feedwater header isolation valves 20, 21, and 22, and main feedwater regulating valves FCV 456, 457, and 458.
SCE personnel have partially disassembled the east flash evaporator unit to determine the extent of damage. Figure 8.9 shows the unit with the evaporator heater, flash chamber, and south water box removed. The flash evaporator condenser tubes are visible in this view. Figure 8.10 shows a rupture of one of the evaporator condenser tubes. SCE is continuing its investigation into the damage to the east flash evaporator unit. Helium leak checks were performed on all east feedwater heaters, revealing no leakage beyond that expected from normal operation. The west feedwater heaters will be leak-tested prior to returning the unit to service. The failure of the flash evaporator had no direct safety significance. 8.5 Turbine Breakable Diaphragms (Rupture Disks) . During the event, steam was observed issuing from the low pressure turbine breakable diaphragms. As shown on Figure 8.11, each low pressure turbine has four breakable diaphragms designed to protect the turbine casing from over-pressurization. The diaphragms, made of thin lead, are designed to break if turbine exhaust pressure, normally subatmospheric, reaches 5 psig. The diaphragms are supported against external atmospheric pressure and normally , seal the turbine casing against air inleakage. All diaphragms were intact prior to the November 21 event. ; 8-2
l Four of the diaphragms ruptured during the event, three on low pressure turbine 1 and one on low pressure turbine 2. Rupture of the diaphragms is not considered unusual for conditions existing after a loss of all ac power with continued energy addition into the main condenser and is of no safety significance. 8.6 Reactor Coolant Pump 8 Thrust Bearing Temperature Indication Reactor coolant pump B was started at 05:01 and at 05:09 the thrust bearing high temperature alarm was received in the control room. When the operators checked the reading, the temperature indicator appeared to have failed high. After discussion, the operators decided to accept the indication reading as valid and started the RCPs A and C and stoppcd B. Subsequent investigation o has determined that the temperature detector failed resulting in the high temperature indication. This failure is considered a random failure, not associated with the event. 8.7 Steam Generator 810wdown Isolation On loss of power, the radiation monitors fail in a mcde which isolates the con-tainment building, including steam generator blowdown. After power was restored, blowdown resumed when radiation monitors in the control room were reset. A re-view of the design and operation of the operational radiation monitoring system (ORMS), shows that blowdown isolation functioned as designed. However, the status of steam generator blowdown is not indicated in the control room, and
! the operators did not recognize that steam generator blowdown was re-established j when the radiation monitors were reset.
8.8 RHR Valve Interlock During the event, initial attempts at opening RHR system valves MOV 813 and MOV 834 were unsuccessful. It was assumed that the pressure permissive inter-lock was malfunctioning and the valves were opened by manually depressing the interlock relay. Subsequent investigation had determined that the permissive performed as designed. The procedure was found to be imprecise and information provided in training did not correspond to actual plant setpoints. As a result, the operators did not correctly understand the response of the pressure-permissive interlock.
. 8.9 Event Recording Systems The majority of the plant capability to record data needed to analyze the event
. was not available. During troubleshooting for the ground on the IC 4kV bus, the power to the Critical Function Monitoring Systems general purpose computer (the Technical Support Center computer), the FOX III, was momentarily interrupted, causing loss of the automatic minute-interval storage of plant data. The system was not reset by the operators. Consequently, when the reactor trip occurred hours later, the 25 minutes of pre-trip data automatically printed was for plant '
conditions not related to this event. The system did not provide useful infor-mation until reset after the event at approximately 08:30. - In addition to loss of the FOX III system, nine control room chart recorders lost pows.- to their chart drives for the duration of the loss of station power. j In seven cases, power for the recording pens was not lost. The recorders 8-3
k continued to record parameters values but without the chart drive. As a result, only the minimum and maximum parameter readings are available for the period the power was off. A summary of the recorders affected include: Chart Drive Lost For: Pen Status: All Steam Generator Continued marking, steam flow, feed flow and water level. Pressurizer pressure, Wide range pressure wide and narrow range, indication lost only. level, water temperature. , RCS Cold leg temperature. Continued marking. T ave /T ref. Failed low. These failures limited data available to monitor plant behavior in the Technical Support Center and hampered efforts to reconstruct the event and to evaluate system performance after the event. However, the operators had sufficient instrumentation during the event to follow their procedures and ensure plant safety. The loss of instrumentation and the ability to trend parameters would have become more important if 19.a event had been of longer duration or had involved additional complications. 8.10 Emergency Notification System During the event, the Emergency Notification System (ENS) red phone in the con-trol room rang spuriously while the plant was experiencing electric power prob-lems. Since the spurious ringing apparently coincided with power system trans-ients, a review and investigation of the power supply system of the ENS was conducted. Figure 8.12 shows the power supply arrangement. The root cause could not be determined and is considered as random noise-induced signals. However, this spurious operation distracted control room personnel and led, in part, to mixed communications and confusion between SCE and NRC personnel. 8.11 Safeguards System The automated safeguards access control system had two malfunctions during this event. Nevertheless, operators and security personnel implemented appropriate planned compensatory measures. The Team identified no signifi- . cant problems that operators had in obtaining access to safety-related plant equipment. 8-4
n a i- , l $ B b E a D
~'
l = O O ! s - Oi l s
-lli I
=
5 e 5 E E f- oj E S l !- i 4 i
. 1 -
t, O u) 9 on O O , m i 9 e a w . J
EVAPORATOR HEATER EVAPORATOR CONDENSER
#5 FW HEATER
#4 FW HEATER
/ / /
NORTH SIDE
- VIEW s x
\ #5 FW HEATER DRAIN COOLER FLASH CHAMBER Figure 8.6 Flash Evaporator (Cutaway View)
^ - ----- - - _ _ _ _ _ _ _ __ __ _ _ _ _ _ _ _ _ , __ _ __
1 1 1
". ,.? $2& %.'
ge
- ;- W,.
. n_. -
,f. n
. s.
Q, .> _. p:.7_ ...f ... / - A ;
.. s
.+
,. , .. %g. ,,g6 3 ,; J , ,{ -
e., . . :s -. ,4 .
. .jg 4 - 5:sa*tg,S .. , m ,
a.+,f_j'-);;..~-f,(-lg{_ ,' - y ', dR%*-. y ll* &
;. :. L ;. n. ; ~ 3 . &y'****'*l: l4,l;' + '. , ;
y ; .wp * . v -
. g . * ~ $[.L ?r +];f '
. y . . 3 ' ' '
^
I
& Qf . <
. ; y .:
, ~- A
_ - lt; ) t,. ,.
}, c. .,.-- l
~
1, - s.
.- 8
^
- , - ,+.J'.
4r ' D'
. #Q,E
..~
+1
., *~: ,
n
+,
'y,3 . . . , .
. ~ '
M,, '
~
I Figure 8.7 East Flash Evaporator Unit (Southwest View) l 4 1 i i i I% _ 3 ,
- f ; ,k, -3v3 s
. . - 4.m., ~-
ea.. k
~, ...}.g, m
. 4
,,. ; p.r: .
.P.*
;g ,
3:m . 3 y.' 7v w e ** 3
..' 4,:?.v., e..
..T tf r
(* .,. 3-i
' f. t a .. kg ~
/
- .4 - '
,1 se.1
,.r.!
)
/ g %, s ..;). / i
* *b '
*:i/;% .
/
1
*6 ' . . WQ.t[
,< ,:t .
/
r l Figure 8.8 East Flash Evaporator Unit Showing Shell ' j Failure (South View) ! i l i
i I
' \.
- IOg M e % .
~
+_
E2$.:b
.v= ::::n ,
g I l I i IX. . l{: ,
.. t?'
j ' a, ,, . y:~ I Figure 8.9 Flash Evaporator Unit with Evaporator Heater Flash Chamber and South Waterbox Removed (Northwest View)
~
' .% - w _;*i~ [.* f A._R .Q & _' Q w p . p ._ p
'#. L*
e . g, h 'yw -p yw .
. .a
,~
( *M4 , 7W s < -
\
1 +:' 1
. . U!. '
I Figure 8.10 Failed Evaporator Condenser Tube
)
N jk WG i . I MIGH PRESSURE TUIISINE I RUPTURED RUPTURED LOW PflESSURE 1 TulteIIIE Ulf0AtlAGED RUPTURED l UNDAMAGED RUPTURED t LOW PflESSURE 2 TURelIIE UIIOAMAGED UNDAMAGEO Figure 8.11 Arrangement of Low Pressure Turbine Breakable Diaphragms
l i l 220 KV AUXIUARY TRANSFORMER C 4 4 - 4100V BUSIC 4180V BUS 2C 11C02 12C02 SWGR NO 1 400VAC SWGR NO:2 400 VAC
}
M 11C10 12C10 kg1:02 ky1202 MCC#1 400 VAC MCC#2 400VAC l NORMAL POWER ALTERNATE i SUPPLY POWER SUPPtV - p-- .
-q NC ' $ - "N0 l MANUAL TRANSFER SWITCH 2 4 l (LOCATED IN 4MV ROOMI
-._ _ __ .J TO SCE
" TELECOM. SYSTEM 3MVA EX10E 8ATTERIES
- I W W 400f12E240V TRANS.
2 HGS 5 144AH r j 1 i , g j , RO3
*n I 120 VAC 1 - 8ATTERY i i COMMUNICATION 12 CHARGER 3 OtSTR. PANEL O LORA N r g-p 48V. 50A FUSE PANEL FUSE PANEL NO: 1 NO: 2 120V740V 1e. 3e .
8REAKER toggig PANEL - INVERTER - 120 VAC A BACK UP OC POWER SUPPLY SE"r g------------ i r g i o [ W : O . l RE0 PHCNES BREAKER 80X (5TOTAU l (ENS SYSTEM) 3CE EQUIPMENT lT Atiti EQUtPMENT l Figure 8.12 Power Supply Scheme for the ENS
9 PRINCIPAL FINDINGS AND CONCLUSIONS The event that occurred at San Onofre, Unit 1 on November 21, 1985, was signifi-cant because (a) all inplant ac power was lost for 4 minutes; (b) all steam generator feedwater was lost for 3 minutes; (c) a severe water hammer was ex- . perienced in the feedwater system which caused a leak, damaged plant equipment and challenged the integrity of the plant's ultimate heat sink; (d) all indi-cated steam generator water levels dropped below scale; and (e) the reactor
, coolant system experienced an acceptable but unnecessary cooldown transient.
In addition, other aspects which contributed to the complexity of the event and to the burden placed on the operators included: a rupture in a flash evapora-tor unit; spurious and incomplete instrumentation indications; fire alarms and system actuations; and malfunctions in the automated security equipment. The Team has concluded that the most significant aspect of the event was that five safety-related feedwater system check valves degraded to the point of inoperability during a period of less than a year, without detection, and that their failure jeopardized the integrity of safety-related feedwater piping. The root causes of the check valve failures have not been determined and are still under review by SCE and its contractors. Potential contributors to this problem include inadequate maintenance, inadequate inservice testing, inade-quate design, and inadequate consideration of the effects of reduced power operations. Maintenance records for these valves were either missing or lacked specificity on what was done. Inservice testing records for these valves were inconsistent; the testing procedure was not rigorous; the test acceptance criteria were subjective; the testing frequency was open-ended; and, the tests did not assure detection of the failures found. These check valves and valves of similar design have a history of like failures. Finally, reduced power operations at Unit 1 are now routine because of steam generator tube plugging and sleeving, and the reduced feedwater flow may have increased the susceptibil-ity of check valve components to hydraulic-induced vibration. In addition to this major conclusion on the underlying cause of the event, the Team has made the following related findings and conclusions. There is no significance to the order in which they are presented.
- 1. The primary cause for the water hammer in the feedwater piping was the failure of multiple check valves in the feedwater system. These failures permitted the piping to empty and fill with steam before the motor-operated feedwater isolation valves were closed. Although the steam condensation-induced water hammer occurred in only one feedwater line, the potential existed for water hammer to occur throughout the safety-related portions of the feedwater system.
- 2. The failures of the five check valves in the feedwater system provided a mechanism for potential common made failure of the heat sink provided by the three steam generators. The failed check valves permitted high pres-sure steam and water from the steam generators to flow back to the low pressure condensate system; the backflow carried with it the auxiliary 9-1
l feedwater flow necessary to maintain the heat sink provided by the steam l generators. Operator actions were necessary to stop the backleakage and prevent a more serious sequence of events. 3. Long horizontal runs of feedwater piping with the potential for voiding are particularly susceptible to destructive steam condensation-induced ; water hammers. Further, operators are not provided the means for detecting the voiding of these lines or givcn guidance on appropriate ways to deal with the situation. Design or procedural changes may be warranted. l
- 4. The flash evaporator failed when overpressurized by the discharge flow of -
an operating feedwater pump due to the partial loss of power and a stuck open feedwater pump discharge check valve that should have prevented the backflow. *
- 5. The timing of the five check valve failures could not be ascertained with certainty. The Team concluded that all check valves had failed prior to the event because the missing parts to the valves were not found in the inspected feedwater piping after the event. Noise from the B steam gener-ator feedwater piping, evident to plant personnel since June 24, 1985, supports the conclusion that the feedwater control station check valve in the B feedwater line had failed earlier. The inspection of the steam gen-erators has not yet been completed by SCE.
- 6. The surveillance procedure for testing the check valves in the Inservice Testing (IST) program lacked adequate methods and objective acceptance criteria for determining whether check valves are closed. Thus, although the check valves had been tested within the past year, operators may have misinterpreted the test results. Furthermore, the IST is not designed to detect developing conditions that may lead to the failure of the check valves, e.g., loose disks and stud nuts.
- 7. The NRC had not completed its review of SCE's Inservice Testing Program.
The initial program was submitted in September 1977 and revised in its entirety on January 24, 1984. Disagreement between SCE and NRC on resolu-tion of certain open issues and scheduling problems with NRC's review have substantively contributed to this delay.
- 8. The resolution of the Unresolved Safety Issue, USI A-1, " Water Hammer,"
did not specifically address the prevention and mitigation of the conse- , quences of condensation-induced water hammers in feedwater piping upstream of the feedring. Interviews of NRC staff involved in resolution of water hammer issues failed to develop citable references, decisions, or discus-sions that provided a basis for excluding further consideration of feed-water piping water hammer. However, in the regulatory analysis of the , l resolution of USI A-1, the staff acknowledged that elimination of water ' hammers is not feasible, that the frequency of water hammers had been sub-stantially reduced by changes in design and operations, and that studies of water hammer had revealed a significantly lesser safety concern than previously hypothesized. It appears that further consideration of water hammers due to main feedwater line voiding was not pursued due to a lack of reported occurrences in U.S. plants.
- 9. NRC's reliance on "J" tubes to delay the development of conditione neces-sary to support steam generator water hammer implicitly assumes that feed-water check valve integrity would be maintained to prevent steam generator 9-2 1
feedring voiding. However, corresponding regulatory requirements to ensure that these check valves performed this safety function were not part of the resolution of the water hammer issue.
- 10. The root cause for the loss of power was a phase-to phase fault of an electrical cable from auxiliary transformer C to bus 1C. The underlying reason for the cable failure has not yet been determined; however, it appears that the cable may have become wetted by a long-term flange leak from the feedwater system, running above the cable tray.
. 11. The plant is designed to experience an extended loss of inplant ac power on loss of offsite power without safety injection. Operators are required to restore power from the switchyard or to load the diesel generators to , restore inplant power. SCE's Emergency Operating Instructions on loss of ac power lack guidance on how long operators can attempt to restore power from offsite sources before the diesel generators should be loaded follow-ing a loss of inplant ac power, or how long the diesel generators can run unloaded without overheating, if their ac powered radiator fans remain de energized.
- 12. The station loss of voltage auto transfer scheme for establishing the delayed access to offsite power may not have functioned as designed. SCE evaluations are continuing.
- 13. The multiple spurious indications early in the event that a safety injec-tion actuation had occurred, added to the confusion of the situation and unnecessarily increased the burden on the operators. Operators diagnosed plant conditions and appropriately disregarded these indications. The safety injection annuciator will always incorrectly alarm on a loss of ac power. This is a design deficiency. The cause of the spurious indication 1
on both safeguard load sequencer system panels is still unknown.
- 14. The operating staff, with the concurrence of management, did not follow appropriate procedures when troubleshooting the electrical ground. Their actions unnecessarily delayed entry into Technical Specification Action Statement requirements that could require plant shutdown.
- 15. Once the electrical ground was located on the feeder from auxiliary trans-former C to bus IC, the operators did not aggressively pursue isolating the auxiliary transformer. Instead, they opted to leave the transformer energized while technicians performed inspections that did not require the transformer to be energized.
l 16. The operators' actions, after the transformer trip, were consistent with l their training. However, in the Team's judgment, some operators lacked detailed plant knowledge in the following areas: i - Cautions associated with paralleling transformers. Requirements for resetting unit generator trips. The process for operating 220KV circuit breakers. Expected indications and timing of the loss of voltage automatic - transfer scheme. Setpoints for residual heat removal system pressure interlock. j
- Expected indication and meaning of lights on SLSS sequencer panels.
l - Operability of diesel generators with auxiliary transformer C reactor coil bypass breakers removed. 9-3
These deficiencies may be due to inadequate operator training and/or procedures.
- 17. On occasion, some site personnel who generally evaluate plant data lacked a sufficiently inquiring attitude. As a result, certain significant indi-cations of underlying reasons for system response or component performance were not detected until brought to the attention of SCE by the Team. It appears that SCE's process for evaluating and following up events may not be sufficiently thorough and systematic to assure that failed components are detected and adequately explained.
- 18. The status of the steam generator blowdown system is not indicated in the control room. The reestablishment of blowdown when the radiation monitors were reset was not recognized and adversely contributed to the cooldown of the reactor coolant system and to the delay in recovering the steam gener-ator levels.
- 19. During the loss of all inplant ac power, sufficient information was avail-able in the control room to enable the operators to follow their procedures and ensure plant safety. However, control room operators had failed to have the Technical Support Center computer reset following electrical ground troubleshooting activities. This failure disabled the computer's ability to record new plant data and thereby denied the operators access to pre-trip and post-trip trends that would have assisted real time and post event analysis and evaluation. Had the station blackout been of longer duration, or involved additional complications, operator responses and the functions provided by the Technical Support Center could have been hampered by the lack of trend data.
- 20. Station maintenance records are incomplete, difficult to locate and, when available, lack sufficient detail to determine what was done.
- 21. The spurious ringing of the NRC red phone at the beginning of the event has not been explained, but it distracted control room personnel and con-tributed to the confusion in the communications between SCE and NRC.
- 22. ENS communications between NRC and SCE were not effective because: (1) the NRC Duty Officer was not knowledgeable about the unique design of the plant and, therefore, misinterpreted operator responses to questions; (2) commun-ications with the plant were initially limited because statements by plant operators incorrectly implied that sufficient personnel were not available ,
to support the establishment of an open line; (3) NRC asked leading ques-tions and operators sometimes did not correct, and in some cases appeared to confirm, inaccurate information; (4) NRC questions characteristically focused on details rather than on the " big picture"; (5) NRC cluttered the communications channel with repetitive discussions about the sequence of events as additional NRC personnel came on the line to the exclusion of obtaining new plant information; (6) NRC resident inspectors relieved more knowledgeable plant operators as ENS communicators and reestablished com-munications at a location remote from real time plant information; and, (7) plant operators failed to inform the NRC of the declaration of an unusual Event.
- 23. There were two malfunctions of the automated security access control equip-ment; however, site personnel implemented appropriate planned compensatory measures, thereby precluding a safety-safeguards interface problem.
9-4
- 24. There was no significant release of radioactivity.
It must be recognized that this report was compiled prior t( fon of all required inspections and evaluations of equipment involved i /ent. SCE's continuing diagnostic efforts have unearthed additional informatiu.s nearly daily; however, this information has been easily integrated into the Team's understand-ing of the incident and in most cases has confirmed long-held hypotheses on the sequence of events. Future reports from SCE will incorporate the findings of those studies which are not yet complete. O e w 9-5
i 1 1 1
)
i APPENDIX A Memorandum from W. J. Dircks, Executive Director for Operations, to the Commission, November 22, 1985 l I l - L
+
APPENDIX A
&fms#o,, ~
UNITED STATES 8*' c NUCLEAR REGULATORY COMMISSION h $ WASHINGTON, D. C. 20555
\'+ ,/
..... NOV 2 21985
. MEMORANDUM FOR: Chaiman Palladino Comissioner Roberts Comissioner Asselstine
, Comissioner Bernthal Comissioner Zech FROM: William J. Dircks Executive Director for Operations
SUBJECT:
INVESTIGATION OF NOVEMBER 21, 1985 EVENT AT SAN ON0FRE UNIT 1 WILL BE CONDUCTED BY AN INCIDENT INVESTIGATION TEAM (IIT) At about 5:00am on November 21, 1985, San Onofre Unit 1 experienced a loss of an auxiliary transformer. Subsequently, a partial loss of electrical power occurred and the control room lighting was lost. The reactor was manually scramed which resulted in a short-term loss of all AC power. A sizeable, unisolable leak was then identified in the feedwater system which is used to naintain steam generator levels, and other failures were experienced in the plant equipment. .The plant is now in cold shutdown. There were no releases and adequate core cooling was maintained at all times. Because of the nature and complexity of this event, I have requested AE0D to take the necessary action 'to send a five member IIT of technical experts to the site to: (a) fact find as to what happened; (b) identify the probable cause as to why it happened; and (c) make appropriate findings and conclusions which would form the basis for any necessary follow-on actions. The team will report directly to me and is comprised of: Thomas T. Martin,
. Director of the Division of Engineering and Technical Programs, Region I; Mr. Wayne Lar.ning, Chief, Incident Investigation Staff, AE00; Mr. Steven Showe, Chief, PWR Training Branch, IE - Chattanooga; Mr. William Kennedy, Safety
. Operational Engineer, Division of Human Factors, NRR; and Mr. Matthew Chiramal, Chief. Engineering Section, AE00. The team was selected on the basis of their knowledge and experience in the fields of reactor systems, reactor operations, human factors, and power distribution systems. Team members have no direct involvement with San Onofre Unit 1. The team is currently enroute to the site.
A-1
The licensee has agreed to a request by Jack Martin, Regional Administrator, to preserve the equipment in an "as-found" state until the licensee and the NRC Team have had an opportunity to evaluate the event. The licensee has also agreed to maintain Unit 1 in a shutdown condition until concurrence is received from the NRC to return to power. The IIT report will constitute the single NRC fact-finding investigation report. . It is expected that the team report will be is:ued within 45 days from now. 4 W iam J. Dircks xecutive Director for Operations cc: SECY OPE OGC ACRS OPA Regional Administrators A-2 l
APPENDIX 8
, Plant Conditions When Water Hamer Occurred and Estimated Piping Support Loads '
1 O
.I
APPENDIX B This appendix deals with voiding conditions when water hammer occurred at SONGS-1, the refilling process and the estimated water hammer loads that resulted as based on analyses of damage to piping supports. These estimates have been used to develop the findings reported in section 6. PLANT CONDITIONS WHEN WATER HAMMER OCCURRED Estimated Voiding of Loop B The void present in loop B feedwater pipe downstream of motor-operated , isolation valve MOV-20 when water hammer occurred was estimated several ways:
- 1. Sequence of events and volumes
- 2. Hydrodynamic instability
- 3. Evidence of water hammer load at FWS-378 The first method relies to a large extent on operator recollections and fill volume calculations; the second method relies on current theories related to steam-condensation water hammer phenomenon and the flow instabilities leading to steam pocket collapse; the third method relies on calculations of the loads required to plastically yield (i.e. , stretch) the bonnet studs of FWS-378 (the 4-inch check valve in the Loop 8 flow control train). This last method is the most reliable. Determining the void fraction is necessary for estimating steam-condensation water hammer loads. Since the water hammer load that affected FWS-378 was a traveling wave (i.e., " classical" water hammer) the extent to which the bonnet studs were elongated can be used to back calculate the impulse wave and the reflected wave.
Sequence-of-Events and Volumes Method The lower estimate of void fraction can be calculated assuming that:
~
- 1. MOV-20 was closed at 04:55
- 2. AFW flow had increased to 155 gpm* (two AFW pumps were operating)
- 3. AFW flow was reduced to zero at 05:00 and then reset to 41 gpm at 05:00:15
- 4. The feedwater pipe downstream of MOV-20 was not completely voided when MOV-20 closed
- 5. The water hammer occurred at 05:07.
- Based on correcting indicated values from flow calibration data found after the event.
B-1
The total injected AFW is calculated to be 1051 gallons; the piping volume downstream of MOV-20 is calculated to be 860 gallons. These calculations indicate a full line when the water hammer occurred, that is, a zero percent void fraction. The upper estimate of void fraction can be calculated from the following assumptions:
- 1. MOV-20 was closed at 04:55
- 2. The AFW flow was 135 gpm
- 3. AFW flow was reduced to zero at 04:59:30 and restored to 25 gpm at -
04:59:45
- 4. AFW flow was constant at 25 cpm
- until the water hammer
- 5. The feedwater line downstream of MOV-20 was completely empty when <
MOV-20 closed. 1 The volume of AFW injected under these assumptions is 789 gallons, or a liquid volume fraction of 84.5 percent. This method yields an estimated void fraction of zero to 15.5 percent. Hydrodynamic Instability Methods The refilling of the feedwater piping is a transient controlled by hydrodynamic instabilities. Initial refill conditions will be determined by whether or not the pipe " runs full," which is a function of AFW injection flow rates. Experiments have show (Reference 1) that a critical Froude number must be reached for the pipe to run full (see below). This was not the case for 150 gpm injection flow and, therefore, initial AFW injection filled the lower portion of the horizontal feedwater pipes for all FW flow circuits. In other words, steam existed along the top of the entire line as cold AFW filled up the bottom. The simultaneous presence of steam and celder water in the horizontal FW pipe results in mass and energy transfer taking place; this transfer results in condensation on the water surface and the pipe wall. The colder water in the bottom of the pipe acts as a heat sink drawing heat from the top of the pipe; the pipe acts as a conducting fin drawing the latent heat of condensation to the colder bottom where the cold AFW is laying. As steam condenses locally, it is replenished from the steam generator (SG) and hydrodynamic instabilities are set up on the water surface. As the water level in the pipe rises (refilling
- is continuous following closure of MOV-20), a more pronounced surface hydraulic interaction is set up, with transition from stratified flow occurring (Figure 6.1.c). As the void fraction decreases, the backflow of steam corresponding to
- condensation on the water surface and pipe wall will become high enough to result in a transition to slug flow; this condition is called a critical void fraction.
Two correlations available for estimating this critical void fraction as a function of steam flow are the Taitel-Ouckler (Reference 2) and Wallis-Dobson (Reference 3) correlations. These correlations are fundamentally the same and since the Wallis-Dobson correlation has been used in previous NRC studies
*0perator observed value.
B-2 L
e I related to SGWH (Reference 4), this correlation was used to estimate the void fraction existing at the time of water hammer occurrence. A lower estimate of critical void fraction of 4 percent was calculated, (assum-ing condensation only on a semi quiescent water surface). Including condensation on the upper pipe wall in the calculations raised the estimated critical void fraction to 15 percent. Both estimates (Reference 5) were based on hand calcu-lational models, which will overestimate void fraction. (A calculation taking into account time and pipe position transients, including pipe wall heat capa-city effects, would be required to refine these estimates.) It should also be noted that these void fraction estimates are based on the total horizontal pipe length (i.e, the 203 feet of B feedwater line between the vertical elbow next to the steam generator back to FWS-346). s Limiting Load Method The water hammer force was back-calculated at FWS-346 by estimating the force necessary to stretch the bonnet bolts approximately 0.5 inches; that force would require an internal pressure of 16,000 psi. Since water hammer wave reflection will essentially double the load (References 6 and 7), the initial impact pressure from a traveling slug was estimated to have been 8000 psi. This load would correspond to a void fraction of 22 percent. Other calculations related to structural support damage (see above) revealed lower forces and support an estimated void fraction of less than 1 percent when the water hammer occurred. In summary, these three methods arrive at overlapping values of void fraction, with a probable range of between 1 and 15 percent and support the hypothesis that the water hammer occurred just about when the Loop B horizontal piping run was nearing complete refill. Flow Conditions When Water Hammer Occurred For steam condensation-induced water hammer to occur, several hydrodynamic phenomena must precede it. First, the pipe must fill (this is a function of refill rate and whether the pipe will run full), the steam-water interface (which controls the rate of heat transfer between the !, team and cold water), the hydrodynamic flow conditions which change as the pipe fills (see Figure 6.1) from stratified flow to slug flow. Eventually a steam pocket is entrapped, which then collapses, and accelerates a water slug in the direction of pressure inbalance.
~
The SONGS-1 water hammer can be attributed to the refill transient (e.g., the time required to refill the voided lines and existing flow conditions and operator actions. Initially, the AFW flow rate (after the MOVs were closed) was 155 gpm. This flow rate is too low to maintain full pipe flow and can be deduced from Froude number considerations. The Froude number (a dimensionless parameter) is the ratio of fluid inertial forces to gravity effects and has been long used to model wave effects (i.e., the " hydraulic jump" phenomena, cc wave cresting in open channel flow). The Froude number necessary to fully fili a circular horizontal pipe (with some water running ahead of the filled section) is approximately 0.5 (Reference 1). B-3
1 k For an AFW injection flow rate of 155 gpm, the Froude number is 0.13 (averaged over the total FW pipe cross-sectional area) and is 0.02 for an AFW flow rate of 25 gpm. Thus, the FW pipe (which'is horizontal) will not run full during the refill transient (see Figure 6.6) and the variation of AFW injection during the time preceding the water hammer is important. The AFW piping refill transient can be examined simplistically as a filling of voided pipe volumes. During the 5 minutes following closure of MOVs -20,
-21, and -22, more than sufficient AFW had been injected to totally fill loops A and C. However, the horizontal piping run in loop 8 was only 90 percent full (Table B-1) and at this point the operators reduced AFW rate to zero and then -
back to 41 gpm. Loops A and C would have been completely filled in about 3.5 minutes since they < have about 50 percent of the volume of loop 8. Thus, this simplistic fill vol-ume approach correlates well with the fact that operators throttled back AFW at about 05:00 following detection of overcooling of the reactor coolant systems. SGs A and C were receiving cold water and reinitiating cooldown (all three SGs had essentially boiled dry previously). Reducing the AFW flow rate to 41 gpm had several adverse effects on the con-tinued refilling of loop B: (1) reverting the refill process to a quieter (or more gradual) hydraulic condition, (2) allowing the steam void at the top of the pipe to propogate backwards to MOV-20 and (3) allowing the cold AFW to stay in contact with the hot steam for a longer period of time. Following resump-tion of AFW injection, calculations indicate that loop B would have been com-pletely filled at about 05:04, suggesting that a steam bubble may have been trapped during the refilling and that the bubble collapsed later.* The water hammer occurred at 05:07. Examination of flow conditions existing at the vertical upturn elbow region at these low Froude numbers (Fr = 0.03 at 41 gpm) is important to an understanding of local flow conditions just prior to the water hammer. The low Froude num-bers which enhance a quiet filling condition versus slug flow may be a signifi-cant factor in averting water hammer in loops A and C. SCE provided informa-tion (Reference 8) related to air-water tests conducted at Creare, Inc. which were designed to sinulate refilling of the horizontal FW pipe at SONGS-1. These experiments showed that for Froude numbers of 0.02.to 0.12: As the liquid level neared the top of the pipe (void fraction of a few percent) the air gap at the vented end of the pipe is bridged by a single * ! slug and water immediately starts filling the vent riser. Most of the remaining air stays trapped in the pipe when filling is continued. , This quiescent refilling, coupled with rapid bridging of the elbow region to fill the vertical pipe at low void fractions (i.e., 5 10 percent), is likely the reason that loops A and C did not experience a water hammer. At SONGS-1 the steam pocket could have been swept out due to a not perfectly horizontal pipe run. The Creare air-water tests were very carefully run to ensure a perfectly level horizontal pipe run. 1 I
*Uncertanties in the calculation assumptions may have contributed to this result.
B-4 I
The total stoppage of AFW and resumption at 41 gpm when loop 8 was 90 percent full delayed fluid bridging of the elbow and increased the time that steam and cold water remained in contact. Whether total flow stoppage or the significant AFW flow reduction were the critical factors leading to a water hammer cannot be determined without a refined thermal-hydraulic refilling analyses. Nonethe-less, operator actions which delayed the total refilling of loop B enhanced the probability the steam condensation-induced water hammer would occur. The hypothesized loop 8 refill conditions prior to the occurrence of water ham-mer are shown in Figure 6.6. A 0.5-inch gap at the top of loop B horizontal FW pipe corresponds to a void of approximately 2 percent. The actual position of the steam pocket at the time of collapse is unknown. However, pipe displace-ments and loads experienced at the damaged supports suggest that the slug was 4 i formed in the horizontal run of loop B piping just upstream from the vertical piping run and then accelerated upstream. This postulated low void fraction condition is supported by the calculated structural damage loads discussed below. Another point of interest is that conditions conducive to water hammer existed well beyond 05:07. Feedwater leakage was manually isolated at 10:45 on November 21, 1985 (Table 3.1). Although backflow of steam for loops A and C would have been blocked by the closure of the MOVs, the failed gasket in 4-inch check valve FWS-378 continued to provide an open backflow path until manual isolation (via closure of FWS-376) was accomplished. COMPARIS0N OF ESTIMATED LOADS WITH DAMAGE INCURRED Souther California Edison (SEC) provided the Team with the following comparison of estimated water hammer loads with observed and measured damage incurred (Reference 9). Figure 6.62 shows measured piping displacements based on survey data taken following the water hammer incident. Characteristics of Damage Forces Based on the survey of the horizontal pipe displacements between the as-found and the design configurations (Figure B.1), it is obvious that the major force exerted on the pipe is opposite to the direction of normal feedwater flow. This force is caused by the water hammer induced pressure wave propogating in the direction opposite that of the normal feedwater flow. A reflected pressure wave was generated following transmission of the first wave which loaded valves FCV-457 and FWS-378. From Figure B.1 it is observed that the largest displacement is along the
, longest run of the pipe (between support location 100 [H00G] and 140 t [H00K].) This displacement is a result of the large impulse load in this section of the line, coupled with increased line flexibility after the support damage done by the water slug. It is also in this region that the FW piping material incurred an 80-inch axial crack. The crack started on the outside of the pipe and had an approximate 25 percent thru wall penetration [from the outside wall (Figure 6.8)].
Pipe Support Damage l The minimum forces needed to damage some of the pipe supports and the I maximum forces that some intact pipe supports can withstand are estimated in this section. B-5
This estimats provides an upper bound and a lower bound of the force exerted on the pipe during the water hammer event. The pipe support locations and identification numbers for various supports are shown in Figure 6.7. A brief description of the calculated loads and findings is provided below.
- 1. H00A - Based on the fact that the base plate was pulled out from '
the wall and the welded dummy support (to which the snubber is attached) was not damaged, it is estimated that the force parallel to the snubber axis was between 40,000 to 90,000 lbs. This trenslates into a force of 45,000 lbs to 101,000 lbs. < parallel to the pipe axis.
- 2. H006 - Based on the fact that the snubber was functionally ,
tested and found to be damaged, it is estimated the force exerted parallel to the axis of the snubber is 40,000 lbs. This translates into a force parallel to the pipe of at least 44,000 lbs.
- 3. H00J - Based on the fact the dummy support was sheared from the pipe, the force parallel to the pipe is estimated between 62,000 lbs. and 164,000 lbs., depending on the manner of loading.
- 4. HOOK - Based on the severe damage of the pipe guide, the force needed to deform the structure is estimated to be around 112,000 lbs. parallel to the axis between support locations 150 and 170.
However, because the energy loss in collapsing the structure and creating the dent is not considered in the analysis, the actual force may be as high as 180,000 lbs.
- 5. HOOL - Based on the fact that the dummy stub was sheared off and the various base plates were pulled off or damaged during the event, the minimum force needed to cause such damage ranges from 36,000 to 82,000 lbs, depending on the nature of the loading.
Timing History Analysis The time-history analysis is another way to estimate the force exerting on the pipe. This method is an iterative process, starting with an estimated force and then comparing the calculated horizontal pipe displacement with ' the observed horizontal displacement. The solution is the force at which i the calculated displacement is close to the observed displacement. The time-history analysis performed by Bechtel (Reference 9) showed that the impulse force, which resulted in a 12-inch displacement of the long north-south run of pipe, is about 160,000 lbs. Analysis Results Based on the time-history analysis and the observed damage to support J HOOK, it is clear that the force exerted on the segnent of the horizontal pipe between the locations 150 and 180 ranges from 112,000 lbs. to 180,000 lbs. Also, the maximum force exerted on the pipe segment, to which the supports H00A and H006 are attached, is limited to 101,000 lbs. Based on B-6
I the experiments done by Swaffield and Phil (Reference 10) the pressure wave transmits only 90 percent of its strength downstream of a 90 degree pipe elbow. Also, the pressure wave tends to decrease as the energy of the impulse is dissipated during propogation. Therefore, the forces exerted on H00A, H006, and HOOK are consistent and are probably caused by the same pressure wave propogating opposite to the direction of the feedwater flow. ' Assuming the energy loss due to damage on the pipe support system between the locations of pipe support HOOK and check valves FWS-346 is negligible,
, the force at the location of FWS-346 can be estimated by increasing the force at the location 160 by 10 percent. This 10 percent is again based on the data discussed in Swalfield and Phil (Reference 10). This increase , will result in a force of 123,000 to 198,000 lbs. at FWS-346. The corre-sponding impact pressure for this force ranges from 1600 to 2600 psi.
Using the methods developed by the CREARE in NUREG-0291 (Reference 4) for NRC's prior water hammer analysis, an impact pressure of 1600 to 2600 psi would be equivalent to a void fraction of 1.5 percent to 2.5 percent, as-suming the subcooling of the slug formed by steam-condensation was in the range of 85 F to 200 *F. The data related to the dents around the first elbow of the horizontal pipe (at the NE corner) and around pipe support location 140 were not used in the above analysis because of the uncertainties in determining the force needed to cause indentation resulting from impact against the concrete. This large uncertainty it composed of various analysis uncertainties re-lated to pipe property, force direction, plastic deformation model, and concrete characteristics before and after the impact. Therefore, based on this more detailed analysis of the damage forces calculated at various damaged pipe supports, it was concluded that:
- 1. The force exerted on the pipe between support locations 40 and 90 ranges from 45,000 to 101,000 lbs. The actual force was probably closer to 101,000 lbs. than to 45,000 lbs.
- 2. The force exerted on the pipe between locations 150 and 180 ranges from 112,000 to 180,000 lbs.
i
- 3. The impact pressure of a water slug, or the surge pressure of i the pressure wave, is estimated to range from 1600 to 2600 psi.
- 4. The void fraction in the main feedwater line at the time of water hammer is estimated to range from 1.5 to 2.5 percent, depending on the subcooling of the water slug. This void fraction is estimated for a total FW pipe length of 203 feet.
l l These calculations further refine the void fraction estimates discussed above l and indicate that very low void fractions existed in the horizontal line just l upstream of the vertical elbow leading to steam generator B (i.e., 1-2 percent void). l B-7
Table B.1 SONGS-1 Feedwater Flow System Parameters and Volumes Design Parameter Data Feedwater Pipe Schedule (Loops A,B,&C) 10-inch, Schedule 60 Inside Diameter 9.75 inches Estimated Length of Horizontal Pipe: l Loops A & C 118 feet Loop B 221 feet . Estimated Volume of Horizontal Pipe: Loop A 459 gal. ' Loop B 857 gal. , Estimated Volume of Vertical Pipe Rise at SG B 60 gal. Estimated Refill Characteristics Loop B: 04:55 to 05:00 @ 155 gpm = 775 gals > 857 gal. Est'd void fraction @ 05:00 = 10% in Horizontal Pipe 05:00 to 05:07 @ 41 gpm = 236 gal. Total Volume Injected = 1011 gal. > (857+60); pipe is full Loop A or C: 04:55 to 05:00 @ 155 gpm = 775 gal. > 459 gal.
> 459+60 gal.
F s B-8
References
- 1. Wallis, G. B., et al., " Conditions for a Pipe to Run Full When Discharing Liquid into a Space Filled with Gas," Journal of Fluid Dynamics, June 1977, pp. 405-413.
- 2. Y. Taitel and A E. Duckler, "A Model for Predicting Flow Regime Transitions in Horizontal and Near Horizontal Gas-Liquid Flow," AICHE Journal, Vol. 22, pp. 47-55.
- 3. G. B. Wallis and J. E. Dobson, "The Onset of Slugging in Horizontal Stratified Air-Water Flow," J. of Multiphase Flow, Vol 1, pp. 173-193, 1973.
- 4. J. A. Block, et al., "An Evaluation of PWR Steam Generator Water Hammer,"
NUREG-0291, U.S., NRC, 1976.
- 5. Letter from F. M. Joos (CREARE, Inc.) to C. Chiu (Southern California Edison) on December 16, 1985. Exhibit 427.
- 6. H. Rouse, Engineering Hydraulics, John Wiley & Sons, Inc., New York, 1949.
- 7. V. C. Streeter and E. 8. W' lie, y Hydraulic Transients, McGraw-Hill Book Company, New York, 1967.
- 8. Joos, F. M. (CREARE, Inc.), to Chiu, C (SCEC) letter dated January 6, 1986.
- 9. Bechtel Power Corporation Calculations MC-077-002, " Hydraulic Transient Load on MFW and AFW Piping at Vicinity of SG-8," December 15, 1985.
Exhibit 495.
- 10. Swaffield, J. A. and Phil, M., "The Influence of Bends of Fluid Transients Propagated in Incompresible Pipe Flow," Proc. Instr. Mech. Engrs., pp.
603-14, Vol. 183. d 8-9
l APPENDIX C
, Regulatory Review of the Potential for Water Hammer at San Onofre Unit 1 e
l l
APPENDIX C NRC's concerns related to the potential for water hammer at SONGS-1 date back to 1975. The correspondence summarized below provides a historical perspective of these concerns. ~ 5/13/75 - NRC informs the licensee (Southern California Edison) that the poten-tial and consequence of secondary system water hammer needs to be analyzed; e that uncovering the steam generator feedring and subsequent operation of the auxiliary feedwater system may cause a water hammer; that changes in plant design or operation necessary to prevent water hammers or assure system inte-grity should be identified for NRC evaluation. 7/14/75 - SCE informs the NRC staff that changes in design or operation of the feedwater system are not necessary; that feedwater flow is continued through periods of low steam generator level; that the layout of the feedwater lines external to the steam generators will minimize the magnitude of a water hammer; that auxiliary feedwater flow is not automatically initiated; that analysis of feedwater piping using dynamic forcing functions modeling water hammer phenomena have not been performed. 9/2/77 - NRC informs SCE of the staff conclusion that keeping steam generator feedwater lines and feedwater spargers full should preclude the occurrence of water hammer and that SCE should propose plant design and procedural modifica-tions to minimize probability of SGWH. 12/27/77 - SCE informs staff no modification to plant design is warranted and that administrative controls will be imposed on operations to assure feedwater is added slowly after the feedwater sparger is uncovered. 5/25/79 - NRC requests information on the design of feedwater lines including questions on the history of feedwater line water hammers. 6/18/79 -SCE provides requested information. 7/3/79 - SCE provides additional information on feedwater line water hammer experiences to supplement response of 6/18/79. 8/2/79 - NRC requests that SCE provide additional information pertaining to susceptability of plant to SGWH per telephone conversation. 8/31/79 - SCE provides requested information on susceptability to SGWH and indicates flow meter indication at low flows is not available. 9/12/79 - NRC acknowledges plant operating history does not show that SGWH has occurred at the plant, but requests additional information to provide further assurance that SGWH would not occur in the future and that surveillance procedures would be adequate to detect water hammer or damage from water hammer, if it were to occur. C-1 APPENDIX C
2/14/80 - SCE responded to the NRC request and informed the staff that the steam generator feedwater spargers have been uncovered many times without SGWH; that administrative controls are in place to reduce the frequency of uncovering the spargers; that transient and accident analysis were not affected by these con-trols; that the impact of automating the auxiliary feedwater system would be evaluated; that visual inspections would be conducted if water hammers occur; that there had been no loss of offsite power with the plant operating; and that further evaluations of the potential for an appropriate corrective action for water hammer would be performed. 4/15/80 - SCE confirms discussions with the NRC staff that the evaluation to be performed by SCE of water hammer would not include SGWH, but would include the development of forcing functions for classic water hammers, such as valve i closure and pump start. 4/22/80 - NRC forwards the safety evaluation report (SER) relating to the potential for water hammer in plant feedwater lines and documents the SCE commitment to evaluate further the potential magnitude of water hammers. The SER concludes that the potential for SGWH is sufficiently low to permit con-tinued plant operations. The basis for this conclusion is stated to be a review of operating history and related operational and procedural characteristics of the feedwater system which showed that although conditions conducive to SGWH , have been encountered, SGWH had not occurred. ' 10/16/80 - SCE informs NRC of plans to automate and upgrade the auxiliary feedwater system in response to TMI lessons learned requirements. A discussion of revised administrative controls to prevent SGWH were not included since they were discussed in the correspondence of 2/14/80. 11/25/80 - SCE provided its promised evaluation of the effects of " classic" type water hammers on feedwater piping in the plant. It concluded that classical water hammer has no significant effect on piping stress and support loads and that existing administrative controls are adequate. 3/6/81 - SCE provides an evaluation of the potential for SGWH with the automated auxiliary feedwater system; indicates that uncovering the feedring cannot be prevented; that flow limits are required because conditions conducive to SGWH cannot be eliminated for the auxiliary feedwater system; that new flow meters enabled improved administrative controls on flow rate; and, that auto- ; mation of the auxiliary feedwater system with these controls does not increase t l the probability of inducing SGWH. l 3/82 - NRC issues NUREG-0918, which summarized the resolution of the concern
- I for SGWH at operating pressurized water reactors. It stated, in part, that:
San Onofre 1 has short horizontal feedwater pipe (less than 3 feet) leading to the SG inlet. SGs still use the " unmodified" feedring with bottom-discharge holes. The auxiliary feedwater flow at the plant can only be started manually: this allows the plant operator to feed the SGs with heated main feedwater whenever possible. The staff has accepted the present implementation at San Onofre 1. However, this matter will be reexamined if any SGWHs occur at the plant in the future. C-2 APPENDIX C
(NUREG-0918 referenced the NRC staff SER produced in April 1980.) It is clear from reviewing the prior correspondence cited above, and from followup interviews with NRC staff who were involved in previous SONGS-1 water hammer reviews, that the thrust of NRC concern was directed at the prevention and mitigation of consequences of SGWH and not at preventing gross voiding of the FW lines. SONGS-1 AFW System Evaluations The potential for cold AFW injection to produce SGWH was included in the NRC and SCE water hammer evaluations discussed above and cited in Section 6.
, Enclosure 2 of Reference 22 of Section 6 provides a review of those evaluations and reaches the conclusion that establishing an upper flow rate limit (i.e.,
150 gpm/SG) on AFW (as recommended in Westinghouse's Bulletin 75-7, Reference 23 of Section 6) would prevent too rapid an injection of cold water into the feedring and, therefore, avoid SGWH. Operators were cautioned in the E0Is not to exceed this upper AFW flow limit whenever the steam generator feedring was uncovered so as not to set up conditions for water hammer. In 1979, as a result of NRC's Bulletins and Orders Task Force review of operating reactors following the TMI-2 accident, SCE was requested to provide information regarding AFWS flow requirements at SONGS-1 (Reference 1). Enclosure 1 of Reference 2 discusses the basis for AFWS flow requirements and includes plant transient analyses dealing with the following transients and accidents identified by NRC staff:
- 1. Loss of main feedwater (LMFW)
- 2. LMFW with loss of offsite ac power
- 3. LMFW with loss of onsite and offsite ac power
- 4. Plant cooldown
- 5. Turbine trip with and withcut bypass
- 6. Main steam isolation valve closure
- 7. Main feed line break
- 8. Main steam line break
- 9. Small break LOCA
- 10. Other transient or accident conditions not listed above.
It should be noted that none of these analyses considered the additional compli-cations which might arise from failure of feedwater system check valves, includ-ing the potential blowdown of steam generators or the failure of the low pres-sure piping in the feedwater system. NRC completed review of AFWS automatic initiation and flow indication (TMI Action Plan Item II.E.1.2) for SONGS-1 in 1982 (Reference 3) and approved the AFWS Technical Specifications in 1984 (Reference 4). The Team met with SCE staff on December 13, 1985 (Reference 5), at which meet-ing SCE reviewed the SONGS-1 AFW system in terms of the original designs, up-i grades (i.e., seismic upgrades), TMI Action Plan requirements, etc. These dis-l cussions also delved into prior water hammer occurrences and SGWH evaluations. I Although this meeting did not reveal any significant new information, it again illustrated the belief that SGWH would not occur within the steam generator, since it had never occurred at SONGS-1, despite numerous transients involving uncovered feedrings, and limits on AFW flow rate were sufficient to preclude such an occurrence. C-3 APPENDIX C l
kkf _k kkhkkkkkkgYkkQ hkQ kg Shkg0kgQqhgQg(qQ9Qgl)gQgQgQgQgQgQgQh w 9 TRANSMITTAL 'IO: / / Docunent Control Desk, 016 Phillips 9/35 o ADVANCED COPY TO: / / The Public Document Boan b g DATE: [ D 'N cc: C&R FPOM: SECY OPS BPN UI tac .
~
papers) Attachcx1 are copies of a Cbnmission meeting transcript (s) and related meeting document (s). They are being forwarded for entry on the Daily Accession List and placanent in the Public Ibcument Pocm. No other distribution is requested or required. Existing DCS identification numbers are listed on the individual documents wherever known. Meeting
Title:
3riek h a o b b ok ( e_ d Qkdo_s oS hho
$ LCD FLTting Date: \ ~d % Open Closed DCS Copies E (1 of each checked)
Itan
Description:
Copies Advanced Original May Duplicate ib PDR , Document be Dup
- Copy *
- 1. TRATERIPT 1 1
. When checked, DCS should send a ,
copy of this transcript to the . LPDR for: , u uwwA,s >
- 2. Mgc \\30 I , }
3. q
- 4. *
- 5 Q
3 G G (PDR is advanced one copy of each docununt, *
- Verify if in DCS, and @
two of each SDCY paper.)
- Change to "PDR Available." @
S 5 . Mm m NN Mym mwayMEMNeuwaEwaWNNawaY}}