ML13310A232

Preliminary Rept, Loss of Salt Water Cooling Event on 800310

ML13310A232
Person / Time
Site:	San Onofre
Issue date:	02/28/1982
From:	Ornstein H NRC OFFICE FOR ANALYSIS & EVALUATION OF OPERATIONAL DATA (AEOD)
To:
Shared Package
ML13310A233	List: ML13310A232 ML13317B073 ML13317B074
References
TAC-65149, NUDOCS 8204090267
Download: ML13310A232 (56)

Text

PRELIMINAR Y SAN ONOFRE UNIT 1 LOSS OF. SALT.WATER.COOLING.EVENT ON MARCH 10, 1980 by the Office for Analysis and Evaluation of Operational Data February..1982 Prepared by:

Harold L. Ornstein NOTE:

This report documents results of studies completed to date by the Office for Analysis and Evaluation of Operational Data with regard to a particular operating event.

The findings and recommendations contained in this report are provided in support of other ongoing NRC activities concerning this event. Since the studies are ongoing, the report is not necessarily final, and the findings and recommendations do not represent the position or requirements of the responsible program office of the Nuclear Regulatory Commission.

8204090267 820407 PDR ADOCK 05000206 P

PDR

TABLE OF CONTENTS Page ACKNOWLEDGMENT.

ii EXECUTIVE

SUMMARY

• 1 1.0 SALT WATER COOLING SYSTEM..

3 1.1 General Description of the Overall System.

3 1.2 Description of Specific Equipment.

5 2.0 THE EVENT 6

2.1 Description of the Event 6

2.2 NRC's Immediate Response to the Event.

8 2.3 Discussion of Licensee's Actions During the Event.

9 3.0 FINDINGS AND CONCLUSIONS.

11 3.1 Importance of the Salt Water Cooling System.

11 3.2 Pump and Valve Failure Experience.

11 3.3 Inservice Testing Program for Pumps and Valves..

13 3.4 Contamination of the Instrument Air System..

15 3.5 Technical Specification Requirements.

16 3.6 Actions Taken by Licensee to Prevent Recurrence.

17 4.0 RECOMMENDATIONS.

19 4.1 Single Failure Analysis 19 4.2 Inservice Testing

• 19 4.3 Contamination of Air System.

19 4.4 Technical Specifiction Requirements.

20

5.0 REFERENCES

21 APPENDIX A - Effects of Lossing the Salt Water Cooling System APPENDIX B - Salt Water Cooling System Pump and Valve Failure Experience APPENDIX C - Inservice Testing of Pumps and Valves APPENDIX D -

Instrument Air System Contamination Problems

ACKNOWLEDGMENT Work on this case study was initiated by James S. Creswell of AEOD during 1980. Subsequent to Mr.

Creswell's departure from NRC in 1981, AEOD Lead Engineer Harold L. Ornstein completed additional work and wrote the final report.

EXECUTIVE

SUMMARY

On March 10, 1980 while operating at 100% power, San Onofre Unit 1* exper ienced a complete loss of the salt water cooling system. The salt water cooling system is the ultimate heat sink for the component..cooling water system which cools certain safety-related equipment. The event involved an unlikely triple failure which resulted in operations exceeding the plant's limiting conditions for operation and was later determined to be an abnormal occurrence. The equipment failures were (1) shearing of the south salt water cooling pump shaft (2) failure of the north salt water cooling pump discharge valve to open, and (3) failure of the auxiliary salt water cooling pump air priming system.

During the initial phases of the event, the temperature of the component cooling water system increased by about 160F; however, it remained well below the upper operating limit. There was no radioactivity release or danger to the public.

However, the plant operators' recovery actions did not include a shutdown as required by the plant technical specifications.

A study of this event was made by the NRC Office for Analysis and Evaluation of Operational Data (AEOD). Analyses that were performed by AEOD and others revealed that there are certain times during plant operation during which a sustained loss of the salt water cooling system could cause significant safety related damage to the plant; i.e., shortly after initiation of the plant's residual heat removal system, there is very little time (a few minutes) available for recovery from a loss of the salt water cooling system. As a result of these analyses, the licensee is reviewing the salt water cooling system's vulnerability to single or common cause failures.

The San Onofre Unit 1 facility is a 450 MWe, three-loop Westinghouse PWR located on the Camp Pendleton Marine Base, just south of San Clemente, California. Reactor criticality was achieved in 1967. The San Onofre facility is one of the plants being reviewed in the NRC Systematic Evaluation Program. Southern California Edison Company (SCE) and San Diego Gas and Electric (SDGAE) are the licensees for the facility.

-2 Desiccant contamination of the instrument air system contributed to one of the failures which occurred during the event.

The presence of the desiccant particles was attributed to shortcomings in plant maintenance.

'If not abated, the presence of desiccant particles in the instrument air system could present a common cause failure mechanism for much of the safety-related equipment at San Onofre. The report focuses attention on the vulnerability of safety-related equipment to such common cause failures at any plant.

A gradual degrading of salt water cooling pump operation was indicated by the

-licensee's inservice testing program. However, information concerning various components revealed by the inservice testing program was apparently not acted on effectively until after the pump failed, initiating the event.

The report addresses the issue of inservice testing from both the standpoint of the licensee's program, and the NRC's involvement.

Shortcomings in plant maintenance and operations which preceded and contributed to the event are discussed. The report also includes a discussion of the immediate corrective actions, and the long-term programs that the licensee initiated to correct the deficiencies.

The NRC Office of Inspection and Enforcement cited the licensee for exceeding the plant technical specifications' limiting conditions for operation.

It also noted shortcomings in plant management controls and in testing and maintenance activities which contributed to the event.

AEOD recommends that the NRC increase its emphasis on licensees' inservice testing programs, and that design, surveillance, and maintenance practices associated with instrument air systems receive scrutiny commensurate with failure vulnerability and consequences at all nuclear power plants.

-3 1.0 SALT WATER COOLING SYSTEM 1.1 General Description of the Overall System A schematic diagram of the salt water cooling (SWC) system appears in Figure 1.

The SWC system is an essential two-train, open-cycle system which acts as a heat.ink for a number of essential and non-essential components such as:

0 Reactor coolant pump oil coolers and thermal barriers Shield cooling coils Excess letdown heat exchanger Seal water heat exchanger Sample heat exchangers Residual heat exchangers Charging pump oil coolers Residual heat removal pumps Spent fuel pit heat exchanger

• Recirculation heat exchanger Gas stripper condenser Heat is transferred from these components to the component cooling water (CCW) system which in turn transfers the heat to the SWC system through the CCW heat exchangers.

In its normal lineup the SWC system pumps ocean water through the CCW heat exchangers where it picks up heat from the CCW system and discharges it to the Pacific Ocean.

During normal operation either one of the two independent SWC trains is capable of performing the system's intended function.

However, when the residual heat removal system is first placed in service during plant shutdown both CCW heat exchangers, and hence both trains of the SWC system, are relied upon (although only one train is needed to assure shutdown).

As shown in Figure 1, in addition to the two (north and south) SWC pumps, the SWC system has an auxiliary SWC pump which serves as a backup and has the same flow capacity (4620 gpm design) as the north and south SWC pumps.

The auxiliary SWC pump can be aligned to either CCW heat exchanger.

(The auxiliary SWC pump is normally aligned to the top CCW heat exchanger, but can be realigned to the

0I INTAKE FROM OCEAN SALT WATER COOLING PUMPS SOUTH NORTH SCREEN TO WASH PUMP TURBINE PLANT COOLERS DISCHARGE AND CIRCULATING WATER PUMP DISCHARGE POV-6 POV-5 O-IN' OUT AUXILIARY COMPNENTCROS TIESALT WATER COONE MOV 720A COOLING PUMP "r

COOLING WATER POV-11 TOP DISCHARGE COMPONENT COOLING TO OCEAN HEATEXCHANGER IN OUT COMPONENT MOV 720B COOLING WATER INTAKE DISCHARGE COMPONENT COOLING FROM OCEAN TO OCEAN HEAT EXCHANGER N/C normally closed.

Figure 1 Salt water cooling system.

-5 bottom heat exchanger by local actuation.) Similarily, there are two screen wash pumps, each of which has smaller flow capability (about 1000 gpm each),

which can be aligned to the CCW heat exchangers. Alternate cooling can also be provided by the facility's fire pumps. (Unit 1 has two 1000 gpm pumps; Units 2 and 3 share two 1500 gpm and one 2500 gpm pump.)

Flexible fire hose would be required to connect these pumps to the CCW heat exchangers.

1.2 Description of Specific Equipment The north and south SWC pumps are vertical, turbine type, electrically-driven, submerged pumps rated at 4600 gpm each, (Johnston Pump Model JT-20DC).

Thiese pumps are classified as safety related. They were purchased to a specification which included a 0.5 g earthquake design requirement (Ref.

1).*

The auxiliary SWC pump is a horizontal, electrically-driven pump, rated at 4620 gpm. It is supplied with a vacuum priming system driven by an air eductor supplied by the service air system.

The auxiliary SWC pump was not classified as safety related, and was purchased to commercial grade specifications.

There are two screen wash pumps. They are vertical, electrically-driven pumps rated at 1000 gpm each. The north screen wash pump can be driven by an internal combustion engine which can be started manually upon loss of electrical power.

The screen wash pumps are normally used to wash off debris accumulated on the plant's intake structure screen. They are not safety related, but were purchased to a specification which included a 0.2 g earthquake design re quirement (Ref. 2).

It should be noted that at the time the plant received its operating license it was not subject to the Commission's present, more demanding equipment qualification requirements.

The San Onofre 1 facility is one of eleven plants presently undergoing seismic and equipment qualification reviews as part of the NRC's Systematic Evaluation Program (SEP).

-6 The SWC pump discharge valves (POV-5 and 6) are double-acting, air-driven, butterfly valves which are supplied instrument air by solenoid valves.

There are no check valves in the SWC pump discharge lines; consequently, failure of the idle pump's discharge valve in the open position would allow a reverse flow if the cross-tie valve is open or if the auxiliary SWC pump is in operation.*

2.0 THE EVENT 2.1 Description of the Event

.At.9:15 pm on March 10, 1980 while the plant was operating at 100% power, the shaft on the inservice south SWC pump sheared. The north SWC pump started automatically as designed, but its discharge valve, which is designed to open automatically when the pump starts, did not open.

The discharge valve failed to open due to a failed air supply solenoid 0-ring. The solenoid 0-ring is believed to have failed due to abrasive action of desiccant which had migrated through the instrument air system to the valve. As a result of the discharge valve failure, the north SWC pump was also inoperable at 9:15 pm.

The control operator manually started the auxiliary SWC pump from the control room at 9:20 pm.

At 9:25 pm the control operator and the watch engineer were made aware by the assistant control operator that the auxiliary SWC pump was not providing any coolant flow.. At about that time, the watch engineer and a plant equipment operator cross connected the discharge salt water flow from the screen'wash pumps to the discharge piping at the north SWC pump.

This connection provided sufficient cooling to the bottom CCW heat exchanger to stop the increase in the During a visit to the plant on July 8-9, 1981, AEOD learned that the licensee was considering installation of check valves in these lines.

The licensee's staff noted that an important feature of the present arrangement is that it minimizes the time during which the salt water cooling pumps experience high starting currents.

To date, a final decision has not been made on this modification.

-7 CCW supply temperature (660F to 820F in ten minutes).*

The screen wash pump flow brought the CCW temperature down to a new equilibrium value of 700F.

At 9:56 pm the control room operators successfully restored the auxiliary SWC pump and aligned it to deliver flow to the top CCW heat exchanger. Because that heat. exchanger was not receiving CCW flow at the time, the auxiliary SWC pump flow had no immediate effect on CCW temperature. In anticipation of a unit shutdown, the watch engineer directed a load reduction from full power beginning at 10:00 pm.

After reducing the unit load from full power by about 3 MW, and after discussing the situation with the supervisor of plant operations, the watch engineer countermanded his earlier order and stopped the load reduction. At 10:13 pm, the top CCW heat exchanger was placed in service, thereby enabling the lauxiliary SWC pump flow to remove heat from the CCW system.

At that time, the watch engineer and the unit superintendent authorized a Plant Change Notice (PCN) to the emergency procedure which had been in effect since 9:15 pm (S-3-5.34, "Loss of Salt Water Cooling to the Component Cooling System").

The change notice diminished the actions required by the licensed operators on loss of salt water cooling. At 12:05 am on March 11, 1980, the discharge valve on the north SWC pump was made operable, thereby concluding the event. Throughout the event the unit was maintained at or near full power.

A week after the loss of salt water cooling event, a thrust bearing on the south charging pump (which ran hotter during the event than usual) was found to be unserviceable and was replaced (Ref.

3).

This is the only equipment which is suspected of having been affected by the event.

No other equipment appeared to be affected by the higher than normal CCW temperatures that took place during the event.

The high temperature alarm setpoint is 97*F.

-8 2.2 NRC's Immediate Response to the Event The NRC was not notified of the loss of salt water cooling event until the next day.

(Notification was made orally to NRC's resident inspector.)

Requirements for "Red Phone" notification per 10 CFR 50.72 went into effect approximately four days prior to the event; however, plant personnel said they were not aware of NRC's early notification reporting requirements at the time of the event.

The resident inspector initiated an investigation promptly. On March 14-15, 1980, he and an NRC regional inspector interviewed the licensee's staff who were involved in the event.

On April 3, 1980, at the conclusion of the investigation, a meeting was held between the licensee's management and the NRC Region V director.

On April 4, 1980, Region V issued an immediate action letter documenting the understandings reached during the previous day's meeting (Ref. 4).

The immediate action letter addressed the licensee's interpretation of plant technical specifications; operator training; review of emergency operating procedures, plant/surveillance, and maintenance programs; and upgrading of the auxiliary SWC pump.

As a result of its investigation of the event, NRC's Office of Inspection and Enforcement (IE) issued an inspection report on April 21, 1980 (Ref. 5).

That report cited the licensee with the following two infractions regarding noncompliance with the technical specifications and failure to follow established emergency operating procedures:

(1) Contrary to plant Technical Specifications 3.3.1A(1)h and 3.3.18, the reactor was operated at or near 100% of rated power for 41 minutes with two SWC pumps and the auxiliary SWC pump inoperable. (During the first ten of those 41 minutes there was no salt water cooling at all.

After the first ten minutes the screen wash pumps were cross connected into the system, and they provided salt water cooling to stop the rapidly rising

-9 CCW temperature).

Also, for over two hours the reactor was operated at or near 100% of rated power with the auxiliary SWC pump operable, but with both SWC pumps inoperable.

(2) Contrary to plant Technical Specification 6.8.1 and station emergency operation instruction S-3.5.34 Rev. 0, Section 3, the reactor was not tripped and continued operating near 100% of rated power when both SWC pumps were inoperable, and when the auxiliary SWC pump was not providing flow to the salt water cooling system.

2.3 Discussion of Licensee's Actions During the Event In retrospect, the plant staff's decision not to shut down the reactor during the loss of salt water cooling event is perhaps understandable when one Considers the licensee's interpretation of the plant technical specifications, (i.e.,

for system operability requirements, the licensee assumed that the auxiliary SWC pump was equivalent to a SWC pump).

It is clear that during the first ten minutes after the pump shaft failure there was no salt water flow to the CCW heat exchangers.

However, plant personnel diagnosed the problem and took actions to provide a backup flow.

Several unsuccessful attempts were made to start the auxiliary SWC pump.

Plant personnel succeeded in valving in the screen wash pump discharge to the CCW heat exchangers to provide enough flow so that the temperature rise of the CCW system was stopped, and its temperatures returned to close to initial (pre-event) values.

Within 45 minutes after the SWC pump shaft failure, the auxiliary SWC pump was restored, and shortly afterwards the CCW system's temperatures returned to normal values.

-10 Regardless of the interpretation of the plant technical specifications, the plant emergency operating instruction (Ref. 6) clearly stated that if both SWC pumps become inoperable, and the auxiliary SWC pump cooling is inadequate, the plant should be shut down.

While the SWC pumps and the auxiliary SWC pump were inoperable, the supervisor of plant operations was contacted by the watch engineer, but neither the supervisor nor the watch engineer ordered a plant shutdown.

The watch engineer and the control operator discussed these conditions shortly thereafter, and apparently

-recognized that the procedure called for a reactor trip.

Discussions with the supervisor of plant operations during a meeting at NRC headquarters that took place on October 30, 1980 (Ref. 7) indicated that.the watch engineer was concerned about tripping the reactor because heat loads would increase on the CCW system. In retrospect, this concern was unfounded, since heat loads would not significantly increase in the hot standby condition achieved after a reactor trip. This should not be confused with going on the residual heat removal (RHR) system to cold shutdown.

Going on RHR does greatly increase the heat load on the CCW system. It should be noted that by tripping the reactor the steam generators could have been used to remove the decay heat, which would decrease to less than 5% in about five minutes.

At the October 30, 1980 meeting, the supervisor of plant operations stated that the. watch engineer was concerned with the manpower required to restore SWC flow, and that he was also concerned about the control room manning and the additional work entailed in tripping the reactor.

The shift technical advisor was not directly involved during this event.

The watch engineer for Units 2 and 3 (which were both under construction) served as the shift technical advisor for Unit 1.

3.0 FINDINGS AND CONCLUSIONS 3.1 Importance of the Salt Water Cooling System The SWC system supplies cooling water to the CCW heat exchangers, which in turn cool much safety-related equipment, including the following:

. RHR pump RtR heat exchangers

. Reactor coolant pump oil coolers

. Charging pump oil coolers

. Excess letdown heat exchanger Seal water heat exchanger

. Recirculation heat exchanger The effects of losing the SWC system were examined in detail by AEOD and are reported in Appendix A. If the plant experiences a loss of the SWC system when the plant goes on RHR, there is very little time for corrective action to prevent excessive CCW temperature.

Figure Al (Appendix A) shows that while going on RHR, a loss of salt water cooling results in the CCW system reaching its limiting temperature of 200 0F within three to six minutes.

Our analysis was not extended to postulate the long term consequences of the failure of safety-related equipment subsequent to the heatup and possible boiling of the CCW system.

However, it would appear that such an event could cause significant damage to the plant.

In discussions between AEOD and SCE (Ref. 8), it was noted that prior to AEOD's review of the event, the SWC system was not reviewed to assure that it was single failure proof.

However, as part of the SEP program, the licensee is now reviewing the SWC system to assure that there is no single credible failure which can cause a loss of the SWC system.

3.2 Pump and Valve Failure Experience The history of past pump and valve failure in the SWC system was examined by AEOD and is reported in Appendix B. As discussed in that appendix, maintenance

12 personnel had reported problems with the south SWC pump several months before its fialure. However, effective maintenance was not performed until after the March 10, 1980 event. Similarly, for two months prior to the event, inservice testing (IST) of the south SWC pump revealed that the pump was not operating satisfactorily and that corrective action was necessary (see Appendix C).

Nevertheless, the licensee did not perform maintenance on the pump, and none of the IST program's required corrective actions were taken until after the pump failed.

The licensee's failure to take the prescribed corrective actions when the pump's performance was not within the IST program's acceptance range defeated the intent and purpose of the IST program. Such inaction appears to have been a root cause of the event.

The first relevant case of serious problems with pneumatically-operated valves failing to operate in the SWC system occurrend about one year before the loss of salt water cooling event.

Prior to the event, difficulty had been experienced on occasion in opening the SWC pump discharge valves-and other pneumatically-operated valves in the plant.

The valve which failed to open during the event was last inspected about eight years before. However, the pump discharge valves received inservice tests every three months to verify operability. Such tests did not reveal any impending failures; nevertheless, the north SWC pump discharge valve failed to open during the event. The licensee's IST records did reveal that the measured stroke times for the SWC pump discharge valves experienced significant variations on occasion without apparent corrective action or increased test frequency as required by the plant IST procedure (see Appendix C).

It should also be noted that in August 1981, the licensee implemented a new, comprehensive preventive maintenance program throughout the plant.

(The

13 licensee is planning to implement a computerized preventive maintenance program in the near future.) The new preventive maintenance program was formulated by an outside contractor (Nuclear Utility Services (NUS) Corporation).

This program appears to be an outgrowth of the NRC's February 1979 Performance Appraisal Team (PAT) inspection of the facility. *This effort verifies the licensee's concern for the problem.

3.3 Inservice Testing Program for Pumps and Valves It appears that prior to the March 10, 1980 event, the NRC and the utility paid superficial attention to the IST Program. There is much documentation available showing utility, NRC and national laboratory involvement in the utility's IST program. This is discussed in detail in Appendix C. For the case of San Onofre, and for the cases of many other nuclear power plants, the current IST programs have not received final NRC approval.

Most plants have only interim approvals, circa 1977. Furthermore, there are several other casesr in a

which NRC gave interim approval for licensee IST programs, which were follbwed by critical reviews by NRC inspectors citing inadequacies in the testing programs.

Carrying out the IST program in strict compliance with the licensee's September 1977 submittal might have prevented the loss of salt water cooling event of March 10, 1980. On discovery of pump performance which was in the be "Required Action Range," the pump would not have been allowed to continue 11 operating without repair or a reanalysis of its design requirements.

ra A significant deficiency in the SWC system IST program resulted from the licensee's request and NRC interim approval of an exemption from pump bearing vibration testing. In view of the large number of SWC pump failures, reconsideration of the exemption, or a "detailed review" as mentioned in NRC's December 22, 1977 approval letter, appears to be appropriate nbw.

14 As noted in Appendix C, the IST data for flow and AP for the SWC pumps can be subject to large errors because AP is obtained from tide height (for many tests reference elevation was included incorrectly), and because the 'SWC pump performance curve is very flat in the range where the inservice testing is performed (250 gpm/ft of head).

A review of the IST records indicated that on several occasions SWC system valve performance was stated to be satisfactory even though it was not. Furthermore, increased testing frequency and corrective actions were not taken on many occasions when the inservice testing results indicated that they were necessary.

One of the major problems with the IST program for the SWC system valves is the lack of specified full stroke travel times or reasonable acceptance criteria.

It appears that the IST data for the valve that failed during the event (POV-5) did not indicate a degraded condition just prior to its failure in March 1980. However, the failure of station operation staff to take corrective action when the IST program requirements were not met defeated the intent and purpose of the program.

As a result of the loss of salt water cooling event, the licensee has focused greater attention on the IST program and has taken actions to strengthen the program. Additional staff has been hired for this work. Pump bearing vibration testing has been added to the program. In addition, the licensee is drafting a set of comprehensive procedures that will assist the operators in pinpointing inadequate equipment performance, and will result in timely corrective action. In view of possible measurement problems, an examination of the IST data reduction process and an error or sensitivity study appears to be needed.

15 3.4 Contamination of the Instrument Air System The licensee's examination of the instrument air system revealed that desiccant had migrated throughout the system. A massive blowdown and cleanup operation was necessary to assure that all 130 safety-related pieces of equipment which are connected to the instrument air system would not malfunction due to desiccant or other air system contaminants.

The possibility of common cause failure due to a contaminated instrument air supply has not been analyzed for San Onofre (or for any other plant to our knowledge).

AEOD examined the problem of instrument air contamination as an important causal factor in the loss of salt water cooling event.

The results are detailed in Appendix D. AEOD noted that discovery of the desiccant contamination in the instrument air system is significant.

However, of greater importance is the realization that for well over a year before the desiccant problem was understood, numerous valves were found to be malfunctioning.

The importance of such pre cursors apparently remained unappreciated until after the loss of salt water cooling event took place.

It should be noted that in addition to finding desiccant in the air system, the licensee found red iron oxides, indicative of corroding carbon steel.

Similar to the desiccant, the rust, caused by moisture in the air system, can pose a common cause failure threat to the plant's safety-related equipment.

As noted in Appendix D, instrument air which is provided by the emergency air compressor is not necessarily filtered.

Such operation may pose a common cause failure potential for much safety-related equipment.

The issue of contamination of both trains of safety-related equipment caused by the emergency air supply has been discussed with the licensee. We are presently unaware of any plans for corrective action.

16 Subsequent to the March 10, 1980 event, the licensee has improved the air drying and filtering system and is planning to implement a program for periodic sampling and monitoring of the instrument air system.

Such actions are expected to greatly enhance plant safety.

3.5 Technical Specification Requirements The plant technical specifications which were in effect at the time of the loss of salt water cooling event were deficient with regard to "operability requirements" (i.e., actions which are required when the plant's limiting

-- conditions for operation (LCOs) and action statements cannot be met).

The plant technical specifications indicated that when the SWC system is inoperable, "the reactor shall not be made or maintained critical"; however, the schedule for shutting down the plant was not specified.

In contrast, the plant emergency procedures more definitively stated that the plant was to be TRIPPED when the SWC system was inoperable (Ref. 6).

• Subsequent to the loss of salt water cooling event, the licensee requested, (and was granted) an amendment to the plant technical specifications regarding "Operability." That amendment was consistent with the standard technical specifications which were in place at newer plants on March 10, 1981.

Those standard technical specifications require that in the event that the LCOs and the associated action requirements cannot be met,

"... the unit shall be placed in at least Hot Standby within one hour, and in at least cold shutdown within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />...

More recently, the Commission revised the standard technical specifications such that the operability requirements are less restrictive than before.

In accordance with Reference 9, when the limiting condition for operation and

17 the associated action requirements cannot be met, ACTIONS MUST BE INITIATED WITHIN ONE HOUR to place the plant in:

At least Hot Standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, At least Hot Shutdown within the following 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, and At least Cold Shutdown within the subsequent 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.

San Onofre has requested a technical specification change to invoke this new technical specification (Ref. 10).

However, the NRC has not yet approved this change for San Onofre.

The loss of salt water cooling event was determined to be an abnormal occurrence for several reasons and was reported to Congress (Ref. 11).

It involved a major degradation of essential safety-related equipment during which the plant staff failed to shut down the plant as required by the plant technical specifications. Essentially, there was a total loss of salt water cooling for about ten minutes, followed by a 45-minute period during which an "unqualified" backup system supplied some salt water cooling flow (the amount of which was much less than the plant's design requirement).

At about 54 minutes into the event the salt water cooling system was restored.

The licensee's failure to reduce power during the March 10, 1980 event would be acceptable within the requirements of the new, more lenient standard technical specifications in which the licensee has ONE HOUR to INITIATE action to shutdown the plant. However, based on the analysis of this event, it appears that the new standard technical specifications may be too lenient for some plants.

3.6 Actions Taken By Licensee To Prevent Recurrence (1) The licensee has reviewed the plant's LCOs and emergency operating instructions. As a result, the licensee has revised the instructions to clearly specify time constraints during which required actions must be taken.

18 Licensed operators have received additional training emphasizing the need to fully and promptly implement the requirements specified in the emergency operating instructions. The training stresses the requirement to promptly shut down the reactor when it is operating outside an LCO.

(2) The licensee has undertaken a major overhaul -of the plant's preventive maintenance program. The licensee hired an engineering consultant (NUS) to prepare a detailed computerized maintenance program. (This effort was initiated as a result of an NRC PAT inspection in 1979.)

(3) The plant's entire instrument air system has been blown down. New desiccant has been installed, as has a new air filtration system including instrumentation to measure the pressure drop across the filters.

The licensee's preventive maintenance program will address the condition of the desiccant.

(4) The IST program has been upgraded. SWC pump testing now includes shaft vibration measurement.

(5) In accordance with NRC's Immediate Action Letter of April 4, 1980 (Ref.

4) plant operating procedures have been modified to preclude consideration of the auxilary SWC pump in determining SWC system operability and LC~s.

On August 27, 1980 the licensee submitted a proposed change to the-technical specifications implementing the directives of the immediate action letter (deletion of the auxiliary SWC pump from system operability consider ations); the NRC has not yet responded to this submittal.

(6) The auxiliary SWC pump's priming system has been modified in an effort to improve its reliability. Furthermore, the licensee is planning to include the auxiliary SWC pump in their inservice testing program after all proposed modifications are complete.

19 4.0 RECOMMENDATIONS 4.1 Single Failure Analysis In view of the potential for significant damage to safety-related equipment at San Onofre due to a complete loss of the SWC system while the plant is on RHR, it is recommended that the ongoing efforts of the systematic evaluation program (SEP) focus on single failure vulnerability and consequences for the SWC system and other equivalent service and cooling water systems.

4.2 Inservice Testing The IST program appears to have been neglected both within the NRC and at the plants. NRR/IE should take timely action to reach agreement on each plant's IST program, and provide the licensees with final safety evaluations as appropriate (in many cases they are about four years overdue).

The licensees should be requested to review their IST and maintenance procedures to assure that corrective actions are taken in full compliance with the approved IST program.

4.3 Contamination of Air Systems Review of the desiccant contamination of San Onofre's instrument air system highlights the susceptibility of safety-related equipment to common cause failures due to contaminated air. It is recommended that each licensee be requested to review their experience with air system contamination (contamination by dirt, desiccant, water, etc.), provide an assessment of the safety implications and evaluate their plant's susceptibility to contamination-induced common cause failures attributable to air systems, including possible complications due to contamination, dislodgement and movement during and following seismic events.

NRR/IE should review each plant's operating experience and susceptibility to such common cause failures and, if appropriate, prescribe a course of corrective action.

20 It should be noted that in response to AEO0's concern for an air system problem at another plant (Ref. 12),

NRR/IE have recently formed a working group to identify the generic implications of air system contamination in safety-related components and systems, and to develop recommendations (Ref. 13).

4.4 Technical Specification Requirements As they are presently written, the latest standard technical specifications (Ref. 9) appear to be too lenient for some situations.

It is recommended that in view of the experience at San Onofre, NRR should again review and, where appropriate, modify the standard technical specifications such that the actions required by the licensee match the seriousness of the event, rather than have one blanket requirement which may not be appropriate for a significant number of events. Some distinction concerning the severity of the events should be made, similar to that required for the prompt reporting of events (10 CFR 50.72).

In addition, IE has issued an Information Notice, "Potentially Significant Equipment Failures Resulting from Contamination of Air-Operated Systems" (Ref.

14).

This notice was provided as an early notification of a possibly significant matter, and no specific action or response was requested from the licensees.

21

5.0 REFERENCES

1. Bechtel Corporation, Vertical Pump Data Sheet, "Salt Water Cooling Pump G13 and G13S," Job No. 3246, San Onofre Generating Station Unit 1, Rev. 0, VPS-E31-12/56.

2.

Bechtel Corporation, Vertical Pump Data Sheet, "Screen Wash Pumps G43 and G43S," Job No. 3246, San Onofre Generating Station Unit 1, Rev. 0, VPS E31-12/56.

3. Letter from H. Ottoson, SCE, to R. Engelken, NRC, April 9, 1980, trans mitting LER 80-010.

4. Letter from R. Engelken, NRC, to L. Papay, SCE, April 4, 1980.

5. Letter from J. Crews, NRC, to L. Papay, SCE, April 21,

1980, transmitting IE Inspection Report No.

50-206/80-09.

6.

San Onofre Nuclear Generating Station, Operating Instruction S-3-5.34, Revision 0, "Loss of Salt Water Cooling to the Component Cooling System,"

February 1, 1980.

7. Memorandum, S. Nowicki,

NRC, "Summary of October 30, 1980 Meeting to.

Discuss the Loss of Salt Water Cooling Event at San Onofre Unit 1 and the Effects of Desiccant in the Compressed Air System," January 6, 1981.

8. Telecon from H. Ornstein, NRC, to R. Ornelas,

SCE, September 2, 1981.

9. Westinghouse, Standard Technical Specifications, Section 3/4 "Limiting Conditions for Operation and Surveillance Requirements," September 16, 1980.

10. Letter from R. Dietch (SCE) to H. Denton (NRC),

transmitting proposed change No.

107 to San Onofre Technical Specifications and Amendment No. 101, December 8, 1981.

11.

NRC, "Report to Congress on Abnormal Occurrences July - September 1980," NUREG-0090, Vol. 3, No. 3, February 1981.

12.

NRC Memorandum from C. Michelson, AEOD, to H. Denton,

NRR, and V. Stello, IE, regarding "Immediate Action Memo:

Common Cause Failure Potential at Rancho Seco - Desiccant Contamination of Air Lines," September 15, 1981.

13.

NRC Memorandum from H. Denton, NRR, to C. Michelson, AEOD, regarding "AEOD Immediate Action Memo on Contamination of Instrument Air at Rancho Seco," October 26, 1981.

14.

NRC Office of Inspection and Enforcement, Information Notice No. 81-38, "Potentially Significant Equipment Failures Resulting from Contamination of Air-Operated Systems," December 17, 1981.

APPENDIX A EFFECTS OF LOSING THE SALT WATER COOLING SYSTEM The SWC system supplies cooling water to the CCW heat exchangers. The following equipment is cooled by the CCW system:

(1)

Reactor coolant pump oil coolers and thermal barriers (2) Shield cooling coils (3)

Excess letdown heat exchanger (4) Seal water heat exchanger (5)

Sample heat exchangers (6) RHR heat exchangers (7)

Charging pump oil coolers (8) RHR pumps (9)

Spent fuel pit heat exchanger (10)

Recirculation heat exchanger (11)

Gas stripper condenser During power operation, items 1, 2, 4, 7, and 9 are heat loads for the CCW system. Items 3, 5 and 11 may be heat loads. Approximately four hours after shutdown, items 6 and 8 become heat loads and items 1, 2, and 4 are removed or greatly reduced.

Several concerns can develop if the CCW temperature increases.

Since the CCW is required to cool plant equipment, loss of cooling can affect equipment operation by degrading performance or causing complete breakdown of the equipment.

The licensee supplied the following limits for maximum CCW temperature (Ref.

A-1) for continued operation of equipment:

Reactor coolant pump bearing oil coolers -

120OF Reactor coolant pump thermal barrier -

120 0F Charging pump bearing oil coolers -

1200F RHR -pump and mechanical seal -

150*F Another limit of concern is the 200*F CCW limit for the CCW pumps (Ref..A-2).

At this temperature the pumps may begin to lose the required net positive suction head and the flow produced by them could be seriously degraded and the pumps could be damaged.

A It should be kept in mind that temperature limitations are usually conservatively established in order to protect plant equipment from damage or failure; consequently, when equipment temperatures increase above such limits, continued reliable performance of the equipment becomes questionable, although equipment failure may not necessarily occur. However, for regulatory purposes when a safety analysis shows that a component's prescribed operating temperature limit is exceeded, that equipment is then assumed to be non-functional and no credit is given for its further use in the safety analysis. In Reference A-2, the licensee has stated that the temperature limit of the CCW is 2000F based upon the limiting temperature of the reactor coolant pump bearings and the design of the CCW system.

A review of the licensee's description for the auxiliary coolant system (Ref. A-3) indicates that:

(1) The maximum load on the CCW system occurs four (4) hours after station shutdown, when both RHR heat heat exchangers (both trains) are in service. The CCW system is designed to supply 115 0F cooling water which provides sufficient operating margin below the maximum permissible cooling water inlet temperature of 120 0F for the reactor coolant pumps.

(2) The RHR system is designed so that with both trains in service the temperature of the reactor coolant is reduced to 140'F within 20 hours2.314815e-4 days <br />0.00556 hours <br />3.306878e-5 weeks <br />7.61e-6 months <br /> after reactor shutdown using salt water at 620F. The RHR system is designed to be placed in service approximately four hours after shutdown, when the reactor coolant system pressure and temperature are less than 400 psig and 350aF, respectively.

-Based on this information, the modes of operation of greatest concern for loss of the SWC system are when the RHR system is in operation during normal shutdown and during main steamline break accident conditions. In accordance with NRC's request (Ref. A-4) the licensee performed scoping analyses of plant performance with degraded salt water cooling under accident conditions (Ref.

A-5).

The bottom line of the analyses was that if there is a total loss of salt A

water cooling shortly after the plant's RHR system is actuated (for the case of normal shutdown or a main steamline break), the CCW system would reach its limiting temperature of 200OF within three to six minutes.

Temperature plots for the two cases of total loss of salt water cooling appear in Figure A-1.*

As seen on this figure for the case of cooldown with.two RHR heat exchangers, the CCW would approach 290*F in ten minutes; for the case of cooldown with one RHR heat exchanger, the component cooling water temperature would exceed 2301F in ten minutes. It should be noted that at some point the water in the component cooling water system will begin to boil and release steam through the head tank.

An analysis to determine when this would occur was not done; however, CCW temperatures in excess of 2300 F are considered physically unlikely.

It is apparent that the loss of both trains of salt water cooling shortly after the plant's RHR system is actuated results in the CCW system exceeding its design limit of 200'F within a few minutes and represents a significant, unanticipated happening requiring quick operator action.

In Reference A-6, the licensee indicated that the station operating instructions require that both SWC pumps be in operation prior to commencing RHR system operation. However, no analysis had been done to assure that there is no credible single failure which will disable both trains of salt water cooling.**

Figures A-1, A-2, and A-3 were generated by AEOD by programming the licensee's equations on the Hewlett Packard 2647A computer/plotter.

• • It should be noted that as part of the NRC's Systematic Evaluation Program the licensee is conducting a single failure analysis of safe shutdown systems. In Reference A-6, the licensee indicated that as a result of discussions with AEOD, their analysis will include a careful review of salt water cooling system failure modes.

300 2 RHR HEAT EXCHANGERS 250 1 RHR HEAT EXCHANGER 2

200 CL E

M i:

150 ch 0:

100 E

00 0

10 20 30 Time in Minutes Figure A-1 Component cooling water temperature during a loss of all salt water cooling at the start of RHR cooling.

A Another case of interest, but which is not as severe, is where one train of salt water cooling is lost shortly after the RHR system is actuated (with two RHR heat exchangers in operation). Results for this case appear in'Figure A-2 and indicate that the CCW system reaches a maximum temperature of 150 0F in about four minutes and stabilizes thereafter. It is not known how long the plant can sustain this temperature without equipment damage; nonetheless, operator action could be taken to decrease the heat load to the CCW system* and thus decrease its temperature to more acceptable levels.

- Two additional cases of interest that were calculated included a large LOCA with no salt water cooling and a large LOCA with 1000 gpm salt water cooling (corresponding to the flow of one screen wash pump).

The results of these analyses (Figure A-3) showed that for a LOCA the loss-of salt water cooling does not result in significant consequences; i.e., for a large LOCA with no salt water cooling, the CCW temperature reaches. 120*F in 20 minutes and levels off at 130aF in about one hour. For the LOCA with one screen wash pump operational, the maximum CCW temperature levels off at about 950F in about 25 minutes.

• Decrease the plant shutdown rate, or initiate heat removal through the steam generator(s).

200 150 100 C

O C;

E 0,

U 50 0

10 20 Time in Minutes Figure A-2 Component cooling water temperature during a loss of one salt water cooling pump at the start of RHR cooling (2 RHR heat exchangers).

-n Component Cooling Water Temperature (7)

I F I I IFI 0

0.M 0

Q 0

0 5

00 P

C

$0 S.Q 0

c CL M

0

8-A REFERENCES A-1. Southern California Edison Company, telecopy of response to October 16, 1980 NRC telecopy, October 28, 1980.

A-2.

Letter from K. Baskin, SCE, to D. Crutchfield, NRC, October 8, 1980.

A-3.

San Onofre Nuclear Generating Station Unit 1, System Description Chapter 6, "Auxiliary Coolant System."

A-4.

Letter from G. Lainas, NRC, to R. Deitch,

SCE, August 29, 1980.

A-5.

Design Calculation No.

DC 814-1, "Component Cooling Water/Loss of Salt Water Cooling Pumps," revised July 30, 1980. (Enclosure 1 to Reference A-2.)

-.A-6.

Telecon from H. Ornstein, NRC, to R. Ornelas,

SCE, September 2, 1981.

APPENDIX B SALT WATER COOLING SYSTEM PUMP AND VALVE FAILURE EXPERIENCE Pump Failure Experience In March 1975, the north SWC pump shaft sheared due to fatigue 'from excessive vibration resulting from worn shaft bearings (Ref. 8-1).

The pump was subsequently repaired by the pump vendor.

On January 16, 1978 (Ref. B-2), while preparations were being made to perform routine maintenance on the north SWC pump, an overload relay in the south SWC pump breaker failed. The auxiliary SWC pump was providing the required salt water cooling flow when the relay failed and it continued to do so until the north SWC pump was returned to service (25 minutes later).

A "Request for Equipment Repair" dated January 28, 1978, (Ref.

B-3) noted that the "shaft wobbles excessively" on the south SWC pump. Maintenance records indicate that repairs were made in October 1978, including installation of a new shaft.

Another "Request for Equipment Repair" dated January 13, 1980 (Ref. B-4),

noted "South Salt Water Pump - excessive seal leak off (getting into motor housing)."

The record also noted that "adj. packing, shaft has excessive runout that indicates worn bushing that should be looked into on outage."

(As noted in Ref. B-5, the licensee has attributed the March 10, 1980 south SWC pump shaft failure to vibration caused by worn bearings.)

The SWC pumps are subject to inservice testing (IST) on a monthly basis. Review of IST data for the SWC pumps indicated that the licensee did not take the required corrective action when the salt water pumps were not performing in their prescribed manner; e.g., the salt water cooling pump "Inservice Testing Program Data Sheets" (Refs.

8-6 and B-7) indicated that on January 8, 1980 and B

on February 21, 1980 the south SWC pump was operating in the "Required Action Range."

Even though the data sheets had a supervising engineer review sign off, the licensee did not perform any maintenance on the pump or take any of the IST program's required corrective actions until after the pump failed on March 10, 1980.

As noted in Ref. B-8, there were several failures of the auxiliary SWC pump's priming system prior to the March 10, 1980 event. The failures were associated with gasket and seal leakage and also a faulty air release valve. However no

-modifications were made to this system until after the loss of salt water cooling event.

Valve Failure Experience The first documented case of problems with the SWC system's pneumatically-operated valves failing to operate occurred about one year before the loss of salt water cooling event.

On March 12, 1979 a "Request for Equipment Repair" was filed (Ref. 8-9) noting that "POV-6 fails to open a lot."

(POV-6 is the discharge valve on the south SWC pump.)

On January 19, 1980, a "Request for Equipment Repair' (Ref. B-10), noted:

"Salt water pump POV sticks when called upon to operate" and "So.

pump would not trip automatically found bad contact W-2 switch-installed temporary jumper to trip relay (WS)"; further, "in witnessing operation of valve it hesitated in the intermediate position and returning star wheels (manual override) back to auto position the valve operated quickly and fully in both directions."

3-B Operators reported to the NRC Region V staff (Ref.

B-11) that, prior to the loss of the salt water cooling event, difficulty had been experienced on occasion in opening the SWC pump discharge valves and other pneumatically-operated valves in the plant. During the event an individual who had previous experience with hard-to-open pneumatically-operated valves was sent to open the north SWC pump discharge valve. Operation of the valve's manual override failed to open the valve.

Failure of the north SWC pump discharge valve (POV-5) was attributed to failure of the solenoid's 0-ring. The solenoid's 0-ring was believed to have failed due to the abrasive action of desiccant (Ref. B-12). The licensee has stated that POV-5 was last inspected approximately eight years prior to the March 10, 1980 event (Ref. B-13).

In contrast, the valve manufacturer's maintenance information (Ref. B-14) recommends periodic inspection of the solenoid valve internals to determine wear or damage (the length of time between inspections depending upon the condition of the air flowing through the valve).

Since inception of the IST program in January 1978, the salt water cooling system discharge valves (POV 5, 6 and 11) have been tested every three months.

Review of the inservice test data for POV-5, 6 and 11 from January 1978 through February 1980 did not reveal any impending failures.

4 -B REFERENCES B-1. Letter from J. Haynes, SCE, to D. Crutchfield, NRC, regarding "Failure of the Salt Water Cooling System, San Onofre Nuclear Generating Station Unit 1," July 24, 1980.

B-2. Letter from J. Head, SCE, to R. Engelken,

NRC, February 10, 1978, transmitting LER 78-01.

B-3. San Onofre Nuclear Station, Request for Equipment Repair, Tag No.

4389, January 28, 1978.

B-4. San Onofre Nuclear Station, Request for Equipment Repair, Tag No. 49144, January 13, 1980.

B-5. Letter from H. Ottoson, SCE, to R. Engelken,

NRC, March 24,

1980, transmitting LER 80-06.

B-6. San Onofre Nuclear Generating Station, Inservice Testing Program Data Sheet, Salt Water Cooling Pumps G13A and G13B, January 15, 1980.

B-7. San Onofre Nuclear Generating Station, Inservice Testing Program Data Sheet, Salt Water Cooling Pumps G13A and G13B, February 21, 1980.

B-8. Letter from J. Haynes, SCE, to D. Crutchfield, NRC, regarding "Failure of the Salt Water Cooling System, San Onofre Nuclear Generating Station Unit 1," July 24, 1980.

B-9. San Onofre Nuclear Station, Request for Equipment Repair, No.

67073, March 12, 1979.

8-10. San Onofre Nuclear Station, Request for Equipment Repair, No. 00266, January 19, 1980.

B-11. NRC Region V staff interviews with SCE staff, March 14-15, 1980.

B-12. Letter from K. Baskin, SCE, to D. Crutchfield, NRC, October 8, 1980.

B-13. Letter from J. Haynes, SCE, to D. Crutchfield, NRC, "Failure of the Salt Water Cooling System, San Onofre Nuclear Generating Station Unit 1," July 24, 1980.

B-14. Automatic Switch Company, "Installation and Maintenance Instructions, General Purpose and Explosion - Proof/Watertight Solenoids," Form No. V-5380R3, 1978.

APPENDIX C INSERVICE TESTING OF PUMPS AND VALVES Approval of the Licensee's Inservice Testing Program NRC's requirements for testing safety-related pumps and valves (10 CFR 50.55 a(g)) invoke the ASME Section XI rules for inservice inspection and testing (ISI/IST) of nuclear power plant components to the extent practicable. Prior to 1978, there were no NRC mandated surveillance or ISI/IST requirements applicable to the San Onofre plant.

In September 1977 Southern California Edison (SCE) submitted a program to the NRC to address ISI/IST requirements for many systems and components, including the SWC system (Ref. C-1).

The Commission's response in December 1977 (Ref. C-2) granted interim approval of the licensee's ISI/IST program. It granted relief from certain code requirements, "...

on an interim basis, pending completion of our detailed review..."

The Commission's letter also indicated that the approval was "based on our preliminary review...," and that, "When our detailed review of your September 28, 1977 submittal is complete, we will: (1) issue final approval of your program...."

The NRC has not provided the licensee this final approval.

(Furthermore, it is our understanding that many other plants have received interim approval based upon a "preliminary review" which has not been superceded by a "detailed review" and a "final" NRC approval.)

Based upon this interim approval, IST of the salt water cooling system was implemented at San Onofre on January 1, 1978.

In accordance with the Commission's ISI/IST requirements, the licensee sub mitted revisions to their ISI/IST program in September 1979 (Refs. C-3 and C-4).

The NRC has approved the ISI program (Refs. C-5 and C-6).

However, the

2-C NRC staff has not responded to the 1ST submittals with a formal approval such as through a Safety Evaluation Report (SER).

Because of this lack of NRC response, it is not clear whether the licensee is required to abide by their original IST submittal (of September 1977) which has an interim approval, or by their September 1979 revision which has no approval.

National Laboratory Involvement in San Onofre's Inservice Testing Program Around 1978, in order to review the deluge of licensee proposed ISI/IST programs, the Commission enlisted the assistance of several national labora tories.

Because of manpower shortages and because other items had a higher priority, only a few plants received formal NRC approvals (SERs) prior to January 1981. Most plants have been operating with interim approvals similar to San Onofre's December 1977 interim approval.

San Onofre's original ISI/IST program was reviewed by Brookhaven National Laboratory (BNL). BNL's review was predicated upon San Onofre's original (September 1977) submittal, a May 1978 clarification document (Ref. C-7),

and upon oral statements which were made at meetings held between NRC, SCE and BNL on June 26 and 27, 1978.

In 1979, BNL published an informal report (Ref.

C-8) which made numerous recommendations regarding San Onofre's ISI/IST program.

Some areas of the BNL report were based upon oral SCE commitments which are undocumented.

For instance, on page 17 of Reference C-8, BNL noted that the licensee committed to performing measurement of motor bearing vibration rather than requiring relief; however, there is no formal SCE submittal indicating such a commitment.

NRC approval of San Onofre's IST program appears cloudy when one examines the NRC review of plant operations discussed below:

C IE Involvement in San Onofre's Inservice Testing Program In February 1979, the NRC Office of Inspection and Enforcement Performance Appraisal Branch conducted an inspection of the San Onofre facility. That inspection consisted of a comprehensive examination of management controls over licensed activities, including maintenance and IST.

On February 9 and 16, 1979, at the conclusion of their visits to the plant and corporate headquarters, the inspectors discussed their enforcement findings and other significant observations with the licensee's representatives (vice presidents, managers, and the plant superintendent). At the interviews, the Performance Appraisal Team (PAT) noted shortcomings in IST, surveillance and maintenance practices.

However, the team's comprehensive report was not completed until eleven months after the inspection was conducted.*

Rather than waiting for publication of the PAT report to be issued, the licensee took action it deemed necessary to correct the deficiencies that were discussed at the exit interviews. It appears that because of the absence of a formal PAT report, the absence of a formal submittal by the licensee, and NRC's focusing on important post TMI fixes (lessons learned, bulletins and orders, etc.), there was no NRC followup of the licensee's corrective action prior to the March 10, 1980 event.

The PAT Report which was published in January 1980 (Ref. C-9) resulted in citations in the areas of inservice testing and preventive maintenance.

The primary reason for this delay appears to be the shortage of manpower within NRC/IE as a result of the TMI-2 accident (which occurred one month after the PAT visit to San Onofre).

Shortly after the accident at ThI, two of the three inspectors who participated in the PAT inspection were reassigned to NRC's investigation of the accident. They remained on this new assignment through September 1979. Furthermore, after their TMI investigation was completed one of these two inspectors was reassigned to a branch chief position at another regional office.

Similarly, the performance appraisal reports for Palisades and Peach Bottom were also delayed because the same two inspectors had also conducted PAT inspections at those plants prior to the accident.

C Regarding IST, the citation noted that:

"The licensee had issued no procedures to describe the performance of pump testing such as vibration measurements and bearing temperature measurements as required by 10 CFR 50.55a(g) and the licensee's identified program for inservice testing of pumps and valves." (It is interesting to note that in December 1977 (Ref. C-2), NRR had approved the waiver on pump bearing vibration and temperature measurements and had provided interim approval of the licensee's pump and valve IST program.)

In February 1980 the licensee responded (Ref. C-10) to the citation on IST. The licensee acknowledged that the findings and the citation were factually correct, but that the procedures for IST had been revised during the year since the PAT inspection. As a result, the licensee believed that its new IST procedures were in full compliance with the NRC requirements. (Once again it should be noted that the NRC has not provided formal approval comments with regard to the licensee's IST procedures which were revised subsequent to December 1977.)

Problem Associated with Inservice Testing of the Salt Water Cooling Pumps Review of the IST data sheets for the SWC pumps indicated that on several occasions the licensee's staff may have erred in taking the tide effects into account; e.g., on April 30, May 21, and June 19, 1981, when the tide was below the zero reference point and the level recorded was a negative number, an arithmetical error was made (rather than subtracting the negative elevation from the pump's elevation, a positive number was subtracted).

Consequently the 6P across the pump was erroneously calculated to be smaller when the tide was low than when the tide was high.

5-C Most of the inservice tests of the SWC pumps were conducted in the range of 3300 to 3600 gpm. As seen in Figure C-1, the performance curve for these pumps is very flat in this range. A flow variation between 3000 and 4000 gpm corresponds to a head variation of only four feet.

Therefore, small inaccuracies in tide level are manifested in large changes in pump flow (250 gpm/ft).

Salt Water Cooling Pump Vibration Testing A significant shortcoming of the licensee's IST program for the SWC pumps may

-. have resulted from granting the licensee an exemption from pump bearing vibration testing. In their original IST program submittal (Ref. C-1), the licensee noted that the SWC pump bearings were water lubricated and were not accessible for vibration testing. Their submittal included a request.for an exemption from the vibration testing and reference value determination, "mainly due to the as-built conditions of San Onofre Unit One and operational conditions." In the interim approval letter (Ref.

C-2), the NRC granted the relief requested noting that

"... this relief is based only on the impracticality of selected ASME Code require ments, we have determined that the relief granted neither increases the probability or consequences of accidents analyzed."

Regarding bearing accessibility for vibration testing, ASME Section XI, IWP 4500, Vibration, states that:

At least one displacement vibration amplitude (peak-to-peak composite) shall be read during each inservice test.

The direction of displacement shall be measured in a plane approximately perpendicular to the rotating shaft, and in the horizontal or vertical direction that has the largest deflection for the particular pump installation.

The location shall generally be on a bearing housing or its structural support, provided it is not separated from the pump by a resilient mounting.

On a pump coupled to the driver, the measurement shall be taken on the bearing housing near the coupling; on close-coupled pumps, the measurement point shall be as close as possible to the inboard bearing.

120 DESIGN POINT M

80 E,

400

.9' 40 0

1000 2000 3000 4000 5000 6000 Flow Rate (GPM)

Figure C-1 Salt water cooling pump performance curve.

C While it is true that the bearings in the pump are water cooled and not directly accessible for measurement, it is also recognized that degrading bearing conditions frequently result in excessive shaft vibration which, in turn, is transmitted to the thrust bearing at the top of the drive motor.

AEOD 6iscussions with the SWC pump vendor indicated that the most favorable location for vibration measurement of the SWC pumps is at the thrust bearing on top of the drive motor (Ref. C-11). This position is believed to offer an optimum amplitude since it is the furthest available point from the flexure point (which is at the sole plate).

Subsequent to the March 10, 1980 event the licensee attempted to perform inservice vibration testing on the SWC pumps.

The data obtained using a shaft rider and a displacement monitor lacked reproducibility and was highly suspect.

The licensee undertook a program to review the methods available for measuring shaft vibration on its SWC pumps.

Based on discussions with the Johnston Pump Company and data from the Hydraulic Institute, the licensee concluded that "taking the vibrations at the top motor bearing is the best representative test of the pump's wearing characteristics" (Ref. C-12).

As part of the IST Program, the licensee measures vibration of the thrust bearing at the top of the drive motor.

This data is used in accordance with ASME section XI IST requirements to determine when the pump shaft requires investigation and/or corrective action.

The Hydraulic Institute's Standard Curve (see figure C-2),

the "General Machinery Vibration Severity Chart" (Ref C-13), and other similar references are used as guides to determine acceptable vibration levels.

It should be noted that in discussions between the NRC, its consultants, BNL, and the licensee, the licensee had committed to perform appropriate tests

S 0

6.0 73 60 00 10 20 00 80 M CLC-)

1.0 Frequeny corrispo tioto FrMqwenncy imbcleseI inue o

irin Figure C-2 Hydraulic Institute curve for pump shaft vibration testing.

9-C on the SWC pumps (Ref. C-8).

In their 1979 revision to the IST program (Ref.

C-3), the licensee agreed to perform vibration tests on the pumps.

As previously noted, the licensee has never received formal NRC approval or comments on the IST program revisions subsequent to 1977. Consequently, because of the relief granted in December 1977, it would appear that the licensee was never required to perform SWC pump vibration tests prior to the March 10, 1980 event.

In-Service Testing Requirements for Salt Water Cooling System Valves The licensee's IST program (see Engineering Procedure S-V-2.15, Table in Ref.

C-1) states that the discharge valves on the north and south SWC pumps (POV-5 and POV-6) and the auxiliary SWC pump (POV-11) are.to be tested once every three months. The testing program requires measuring and recording maximum full stroke times for each of these power operated valves.

Furthermore, the requirements (see Engineering Procedure S-V-2.15,Section IV.A.1.C.3 in Ref. C-1) state that:

If an increase in stroke time of 25% or more from the previous test for valves with stroke times greater than ten seconds or 50% or more for valves with stroke times less than or equal to ten seconds is observed, test frequency shall be increased to once each month, if possible, until corrective action is taken, at which time the original test frequency shall be resumed.

In any case, any abnormality or erratic action shall be reported.

The measured stroke times for POV 5, 6, and 11 are listed in Table C-1.

As noted from the table, there were several tests in which corrective action was required by the engineering procedure including an increase in test frequency.

However, the data indicates that in only one case prior to the loss of salt water cooling event (POV-6, during the March 1979 test) was any corrective action noted.

All other tests were listed as satisfactory even though the stroke time requirements were not met.

In no case was the test frequency increased from quarterly to monthly as prescribed by the IST program requirements.

C Table C-1 SALT WATER COOLING SYSTEM INSERVICE TESTING DISCHARGE VALVE STROKE TIMES POV 5 POV 6 POV 11 North SWC Pump South SWC Pump Auxiliary SWC Pump Discharge Valve Discharge Valve Discharge Valve Date (sec.)

(sec.)

(sec.)

1/78 4

7 4

4/78 1.57 11.22 1.22 8/78 7.0 9.5 1

12/78 2

4 2

3/79 2.5 21*

1.5 6/79 2

2 2

9/79 3

3 3

1/80 4

1.5 2

3/80 3

2 2

• Repaired air line to POV-6 stroke time open 2 secs.

APPENDIX D INSTRUMENT AIR SYSTEM CONTAMINATION PROBLEMS A schematic diagram of the compressed air system appears in Figure D-1.

The system consists of three air compressors and receivers that are connected to a common three-inch heater which feeds both the service air system and the instrument air system.

The service air system uses compressed air which is tapped off the common three-inch header and goes directly (without any filtering or drying) to the service air headers for use throughout the plant.

The instrument air system receives air from the common three-inch header which is routed through desiccant column air dryers and filters, and then through the instrument air header to equipment throughout the plant.

In addition to the three main compressors and receivers, there is an emergency air compressor and receiver. Instrument air supplied by the emergency air compressor can go through the instrument air dryers and filters.

However the emergency air compressor discharge can bypass the dryers and filters and feed directly into the instrument air header. The remainder of the-discharge from the emergency air compressor is normally unfiltered and goes to the redundant instrument header and from there to equipment served by both the service and instrument air systems.

The air dryers, through which instrument air is routed, are vertical tanks containing silica gel desiccant material in bulk form. It is believed that, due to the prolonged use between replacement of the air dyrers at San Onofre, the desiccant broke down into small fragments (Ref. D-1).

Some of the fragments

TO INSTRUMENT AIR HEADER INSTRUMENT AIR FILTERS EMRENCY AR COPESSOR ITH INSTRUMENT IVER AIR DRYERS TO REDUNDANT INSTRUMENT AIR HEADER TO SERVICE AIR HEADER AIR RE EIVERS FERT-R FILTER FILTER INSTRUMENT AND SERVICE AIR COMPRESSORS Figure D-1 Schematic diagram of the compressed air system.

D migrated from the tanks to the filters downstream. These filters, which also contained desiccant in bags, were initially effective in removing the small desiccant fragments from the air. However, the desiccant fragments gradually coated the filters, thereby increasing the pressure drop across the filters.

This increased pressure drop caused increased bypass flow around the filters, and the small desiccant fragments eventually appeared downstream. Desiccant fragments then migrated throughout the instrument air system to areas inside containment, the auxiliary building, and the intake structure.

During the loss of salt water cooling event, the pneumatically operated valve (POV-5) on the discharge of the north SWC pump failed to open on demand subsequent to the failure of the south SWC pump shaft. This valve failure blocked the flow of salt water from the redundant (north) SWC pump to the bottom CCW heat exchanger.

The licensee has noted (Ref.

D-2) that there are two predominant mechanisms in which the presence of desiccant can lead to solenoid valve failure.*

The first failure mechanism is one in which the desiccant enters a solenoid core and prevents proper operation. The second involves wear of solenoid components due to the abrasive action of the desiccant on moving parts.

It should be noted that maintenance information from the solenoid valve vendor (Ref. D-2) recommends, "keep the medium flowing through the valve as free from dirt as possible." It also notes that, "In general, if the voltage to the coil is correct, sluggish valve operation, excessive heating or noise will indicate that cleaning is required."

D Failure of the north SWC pump discharge valve (POV-5) was suspected to have been caused by the desiccant's abrasive action on the solenoid's 0-ring.

Sluggish operation or malfunctions of POV-5 and -6 and other pneumatically actuated equipment had been observed due to desiccant in the instrument air prior to March 10, 1980. As noted in Reference D-1, the licensee had taken action to correct a desiccant carryover problem which was discovered earlier (containment isolation valve CV-537, which led to installation of temporary filter pads in the instrument air system in February 1980).

At that time, the licensee apparently was not aware of how widespread the desiccant contamination problem was.

Subsequent to the loss of salt water cooling event (during the April 1980 refueling outage), the licensee became aware of the fact that desiccant had spread throughout the instrument air system. The licensee's cleanup of the instrument air system involved a sequential blowdown and venting of all instrument air lines, and the testing-of all safety-related equipment and instruments which could have been contaminated by the desiccant.

In Reference D-2, the licensee indicated that there are approximately 130 safety-related pieces of equipment, including the pressurizer power operated relief valves and the associated block valves, which are connected to the instrument air system.

The licensee estimated (Ref. 0-4) that between 400 and 800 man-hours were spent in the plant cleaning out desiccant from the instrument air system and related equipment.

The cleanup consisted of starting at the main air header and sequentially blowing down all of the lines (safety related valves where desiccant was found are listed in Table D-1).

AEOD reviewed the documentation associated with the blowdown process and discussed it with the licensee representatives during a meeting on October 30, 1980 (Ref.

D-5).

D TABLE 0-1 SAFETY-RELATED VALVES WHERE DESICCANT WAS FOUND*

Reactor Coolant System CV 532 CV 544 Chemical and Volume Control System PCV 1115A Component Cooling Water System CV 722A RCV 605 Salt Water Cooling System POV 5 POV 6 Miscellaneous Water System CV 150 CV 537

• See Reference D-2.

D The technique for removing the desiccant and the scope of the equipment to be cleaned was not reviewed by the licensee's onsite review committee nor was an approved, written procedure used during the process. However, at a July 8, 1981 meeting, the licensee informed AEOD that, subsequent to the October 30, 1980 meeting, the onsite review committee and the NRC resident inspectors did review the procedures for determining that the air lines (to safety-related equipment and instruments) were free of desiccant and other particulate matter.

In Ref. 0-1, the licensee acknowledged the presence of iron oxides along with desiccant which was removed from the instrument air lines. The red iron oxides found were believed to emanate from oxidation of carbon steel piping in the system, whereas the black iron oxides were believed to be from the original mill scale. Apparently, moisture which was not removed by the desiccant contributed to the corrosion of the carbon steel in the instrument air system.*

Failed or degraded pneumatically-operated valves at the San Onofre plant due to poor air quality have not been limited to the salt water system.

Prior to the October 1978 refueling outage, sluggish valve operation was experienced on the three main feedwater regulating valves. Repair of these valves indicated the presence of "grit" in the air system supplying the valves.

As noted by the licensee in Reference D-1, the source and significance of the grit was not understood.

• On January 9, 1980 a containment isolation valve (CV-537) failed to close during routine testing.(Ref. D-6).

An investigation revealed that gritty In Reference D-4, the licensee indicated that there are no instrument air system cleanliness or moisture specifications and that there is-no periodic air sampling or monitoring of the instrument air system quality.

However, the licensee also indicated that the desiccant air dryers were sized so that when they operated correctly the instrument air system would have a

-200F dew point.

material had entered the solenoid valve and prevented it from operating. The source of the gritty material was discovered to be desiccant from the instrument air dryers.

On July 17, 1980, subsequent to the loss of salt water cooling event, containment isolation valve CV-537 again failed to close on demand. An investigation revealed that desiccant from the air dryers had entered the solenoid valve core and prevented it from operating (Ref. D7).

8 -D REFERENCES D-1. Letter from J. Haynes, SCE, to R. Engelken, NRC, regarding "Instru ment Air System Malfunction, San Onofre - Unit 1," February 23, 1981.

D-2. Letter from K. Baskin, SCE, to.D. Crutchfield, NRC, October 8, 1980.

D-3. Automatic Switch Company, "Installation and Maintenance Instructions, General Purpose and Explosion-Proof/Watertight Solenoids," Form No.

V-5380R3, 1978.

0-4. Telecon from R. Ornelas and S. Scholl, SCE, to H. Ornstein, NRC, August 6, 1981.

D-5. Memorandum, S. Nowicki,

NRC, "Summary of October 30, 1980 Meeting to Discuss the Loss of Salt Water Cooling Event at San Onofre Unit 1 and the Effects of Desiccant in the Compressed Air System," January 6,,.-1981.

D-6. Letter from H. Ottoson, SCE, to R. Engelken, NRC, February 5, 1980, transmitting LER 80-003.

D-7. Letter from H. Ottoson, SCE, to R. Engelken, NRC, August 18, 1980, transmitting LER 80-032.

C It should be noted that one of the major problems in determining the acceptability of valve performance is the fact that the IST program and the station procedures do not specify the full stroke travel time to be expected or prescribe reasonable acceptance criteria based on the accuracy of the test arrangement.

C REFERENCES C-1. Letter from K. Baskin, SCE, to K. Goller, NRC, regarding "Docket No. 50-206, Provisional Operating License No. DPR-13 Inservice Inspection and Testing Programs San Onofre Nuclear Generating Station Unit 1," September 28, 1977, and enclosures (ISI/IST procedures, September 9, 1977).

C-2. Letter from A. Schwencer, NRC, to 3. Drake, NRC, regarding San Onofre Nuclear Generating Station Unit 1, December 22, 1977.

C-3. Letter from K. Baskin, SCE, to D. Ziemann,

NRC, September 4, 1979.

C-4. Letter from K. Baskin, SCE, to D. Ziemann, NRC, regarding "Inservice Inspection and Testing Program, San Onofre Nuclear Generating Station, Unit 1," September 11, 1979.

C-5. Letter from D. Ziemann, NRC, to J. Drake, SCE, transmitting a safety evaluation and Amendment No. 46 to the San Onofre Unit 1 provisional license, September 26, 1979.

C-6. Letter from D. Ziemann, NRC, to R. Dietch, SCE, transmitting a safety evaluation and a modification to Amendment No. 46 to the.San Onofre Unit 1 provisional license, April 30, 1980.

C-7. Letter from K. Baskin, SCE, to D. Ziemann,

NRC, regarding "Inservice Inspection and Testing Program (IST) and Proposed Change No.

67 to the Technical Specification of San Onofre Nuclear Generating Station Unit 1,"

May 26, 1978, and Enclosure.

C-8. Brookhaven National Laboratory, Informal Report, "Recommendations to the NRC for the Safety Evaluation Report of San Onofre Nuclear Power Station, Inservice Inspection and Testing Program, Revision 2,"

BNL-NUREG 26625, July 1979.

C-9. Letter from J. Crews, NRC, to J. Moore,

SCE, regarding "NRC Management Inspection of San Onofre Unit 1," dated January 16, 1980, transmitting Notices of Violation, and IE Inspection Report No. 50-206/79-01.

C-10. Letter from N. Papay, SCE, to R. Engelken, NRC, February 6, 1980.

C-11. Letter from R. Karon, Johnston Pump Company, to J. Creswell, NRC, regarding "Shaft Breakage, Johnston Model 20 DC," June 26, 1980.

C-12. Memorandum to File from D. W. Bailey, San Onofre Nucleat Generating Station, regarding "Inservice Vibration Testing of Salt Water Cooling Pumps at San Onofre -

Unit 1," April 26, 1981.

C-13. To be furnished later.