IR 05000206/1985033
| ML13323B055 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 10/30/1985 |
| From: | Dangelo J, Huey F, Johnson P, Stewart J, Tang R, Targ R, Tatum J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML13323B053 | List: |
| References | |
| 50-206-85-33, 50-361-85-32, NUDOCS 8511200221 | |
| Download: ML13323B055 (12) | |
Text
U.S. NUCLEAR REGULATORY COMMISSION
REGION V
Report Nos. 50-206/85-33 and 50-361/85-32 Docket Nos. 50-206 and 50-361 License Nos. DPR-13 and NPF-10'
Licensee:
Southern California Edison Company P. 0. Box 800, 2244 Walnut Grove Avenue Rosemead, California 91770 Facility Name:
San Onofre Units 1 and 2 Inspection At:
San Onofre, San Clemente, California
'Inspection Conducted:
September 19 - October 11, 1985 Inspectors:
to a
F. ey, Senior Resident Date Signed Inspe or, Units.1, 2 and 3 rJ. P. Stewart. Resident.Inspector Date Signed C D'Angelo, Resident Inspector Date Signed
-.
Tatum, Resident Inspector Date Signed
_4rR. C. Tang, Resident Inspector Date Signed Approved:
____
P.. H ohnson, Chief Date Signed Reac Projects Section 3 Summary:
Inspection on September.19 -
October 11, 1985 (Report Nos. 50-206/85-33 and 50-361/85-32).
Areas Inspected:
This is the report of a special inspection to review the circumstances
.involved with the failure of a Unit 1 auxiliary feedwater pump. The event occurred on September 19, 1985, when the pump started automatically in response to low steam generator water level following an inadvertent plant trip. Related activities and safety related components in Unit 2 were also examined. The inspection involved 99 hours0.00115 days <br />0.0275 hours <br />1.636905e-4 weeks <br />3.76695e-5 months <br /> on Unit 1 and 75 hours8.680556e-4 days <br />0.0208 hours <br />1.240079e-4 weeks <br />2.85375e-5 months <br /> on Unit 2 for a total of 174 hours0.00201 days <br />0.0483 hours <br />2.876984e-4 weeks <br />6.6207e-5 months <br /> by five NRC personnel, including 25 hours2.893519e-4 days <br />0.00694 hours <br />4.133598e-5 weeks <br />9.5125e-6 months <br /> of backshift/weekend inspection effort. Inspection Procedures 93702, 71707, 61726, 62703, and 71710 were covered during this inspectio pDR AfDOCK OO~
-2 Results:
Two apparent violations were identified:, (1) Failure to comply. with Technical Specification requirements to properly review and control modifications to safety related equipment, and (2) Failure to comply with station procedure requirements to properly report, document, and initiate appropriate corrective actions for observed deficiencies-on safety related equipmen DETAILS Persons Contacted
- H. Ray, Vice President, Site Manager
- H. Morgan, Station Manager
- M.. Wharton, Deputy Station Manager
- D. Schone, Quality Assurance Manager D. Stonecipher, Quality Control Manager
- R. Krieger, Operations Manager
- D. Shull,. Maintenance Manager
- J. Reilly, Technical.Manager
- B. Zintl, Compliance Manager J. Wambold, Training Manager
- W. Marsh, Operations Superintendent, Units 2/3
- J. Reeder, Operations Superintendent, Unit 1 V. Fisher, Assistant Operations.Superintendent, Units 2/3
- B. Joyce,.Maintenance Manager, Units 2/3 H. Merten, Maintenance Manager, Unit 1 D. Sheridan, Maintenance Supervisor, Unit'l
.R. Santosuosso, Instrument-& Control Supervisor T. Mackey, Compliance Supervisor
- C. Kergis, Compliance Engineer San Diego Gas and Electric Company
- R. Erickson, Senior Engineer
- Denotes those attending the exit meeting on October 11, 1985.
The inspectors also interviewed and talked with other licensee employees during the course of the,,inspection; these included shiftjsupervisors, control room operators, :maintenance personnel and Quality Assurance personne..
Summary of Event:
On September 19, 1985, the Unit 1 steam turbine driven auxiliary feedwater (AFW)
pump trippedas a result of failure of both journal bearings on the pump turbine. The bearings failed as the'result of an undetected loss of lubricating dil from one of the bearing sump The specific sequence of events involved in this incident was as follows:
September 19, 1985 1825 -
Operations approval was granted to a maintenance electrician to add nitrogen to the Unit 1 "A" and "B" auxiliary transformer The reactor tripped as a result of a turbine trip. The turbine tripwas caused by an inadvertent protective trip of the "B"
auxiliary transformer, due to ajmomentary high-transformer nitrogen pressur The turbine driven AFW pump started-automatically in response to low steam generator level, resulting froni the turbine tri The turbine driven'AFW pump tripped as-the apparent result of a mechanical overspeed trip actuation. This trip was later-determined to be-the iesult-of'exkessive turbine vibration
.,caused by damaged bearings.
1957
-
The overspeed trip was,reset on the turbine driven AFW pum A Limiting Condition for-Operation Action Requirement (LCOAR)
was initiated in accordance with 'Technical Specification action statement -3.4.3.B, pending arrival of maintenance and cognizant engineering personnel to evaluate the cause of the pump tri NRC notification was made in accordance with 10 CFR 50.72(b)
(2). (ii).
- 2232
-
Engineering and maintenance personnel confirmed failure of the pump turbine bearing September 20, 1985 1125 -
A reactor'plant cooldown was starte Entered Mode 4 Safety Significance of Event a. - Design/Operational Significance (1) The turbine driven AFW pump is one of two independent safety related pumps required by technical specifications to ensure a heat sink for reactor plant cooldown in the event of a total loss of offsite powe (2) Loss.of the turbine driven AFW pump during Mode 3 operation leaves one electric driven pump to feed the steam generators for plant cooldown in the event of a total loss of offsite powe (3) During this.event there was -no loss of offsite power and both the main feed pumps.and motor driven AFW pump were available to feed the generator (4) During this event no technical specification Limiting Conditions for Operation-(LCO) were exceede Management Significanc This event is of, management significance in that it demonstrated basic, repetitive errors in the manner in which important safety related plant components were.being maintained. As noted in the following sections of this report, both supervisory and craft maintenance personnel, on separate occasions, violated fundamental maintenance administrative procedures which were clearly written and on which they had been recently trained. This indicates that although the licensee.has developed and implemented a comprehensive program for ensuring proper maintenance of safety related equipment, station management has not been effective in:assuring that maintenance personnel understand management's expectations with regard to the need to adhere to established written policies and procedure.
Scope of NRC Revie The NRC inspectors conducted an independent review of the circumstances leading up to and associated with this incident. The scope of this review included the following: Review of control room log Review of maintenance orders associated with the turbine driven AFW pum Review of deficiency tags issued against the turbine driven AF pum Review of recent QA/QC findings with regard to maintenance-related activitie Review of station operating, maintenance and vendor procedures dealing with the operation, surveillance and maintenance of safety related equipmen Inspection of the damaged AFW pump immediately after the event and during subsequent maintenance effor Interviews with station management, operations and maintenance personnel involved with the AFW pump Inspection of other safety related pumps with similar bearing lubrication system.
Results of NRC Review Events Preceding Incident On August 6, 1985, maintenance personnel changed the turbine driven AFW pump.oil under Maintenance Order (M.O. #85052574).
During the performance of this MO, the inboard and outboard bearing oil level
II)'-
sight glasses were:found to be leaking and a deficiency tag was hung by Nuclear Quality Control (NQC).
Maintenance personnel performed additional corrective maintenance on the leaking outboard bearing sight glass in conjunction with the above MO. This work was not authorized by the MO, nor was its performance recorded on the MO. Specifically, during performance of this MO, it was noted that the oil sight glass on the outboard turbine bearing did'not change level as oil was added to or drained from the bearing. The sight glass was removed for inspection and it was found that the sight glass tube was plugged at the top with a putty-like material which had solidified, preventing proper ventin The glass tube was cleaned and reassembled. After reassembling, it was noted that the sight glass was leaking from the bottom of the glass tube. A maintenance supervisor determined that the leak was caused by a lack of pressure on the.gasket under the glass tub Using a piece of foam rubber ear plug, he fashioned a compression piece for the top of the glass tube and installed it into the sight glas On August 9, 1985,' maintenance performed additional work on the turbine driven AFW pump under another M.O. (#85080655) to fix an oil leak from the drain plugs of the inboard and outboard bearing Maintenance drained the outboard bearing housing to the correct oil level, and cleaned and reinstalled the drain plug. The inboard bearing drain plug was tightened and checked for oil leaks around the drain plug. It was again noted that the inboard and outboard bearing oil level sight glasses were leaking. Two additional deficiency tags.were hung because of the leaks. No other oil leaks were observed at that.tim The licensee determined that.after August 6, 1985 but prior to the event on September 19, 1985, operations personnel had added oil to the inboard bearing about 20 times (17 times through the sight glass and 3 times through the casing plug). Oil was not added to the outboard bearings, since the oil level at the sight glass appeared satisfactory. However, operations personnel had reportedly wiped oil from underneath the sight glass of the inboard bearing twice and underneath the outboard bearing 3 times during this perio Operations personnel also noted that the outboard bearing sight glass leak appeared to be fixed, but the inboard bearing sight glass was still leaking. Operations performed operability tests on both feedwater pumps in accordance with S01-12.3-26 on August 6, 1985 and on September 3, 198 The results were satisfactor After August 6, 1985, Station Engineering performed in-service inspection testing (IST) twice on the turbine driven and electric driven AFW Pumps. The tests were verified to be satisfactory. The last test was conducted on September 3, 1985. Vibration levels were also reported as satisfactor On September 19,-1985, after Unit 1 tripped on sudden pressure in the Auxiliary "B" transformer, the turbine driven AFW pump, which started on low steam generator level, tripped apparently on
overspeed. Engineering and maintenance personnel were called in to perform testing to assure pump operability. After 'the AFW pump ha tripped, engineering and maintenance personnel verified via the sight glass that oil levels' of the inboard and outboard bearings were satisfactory and that the overspeed trip had'been reset. The pump was restarted for testing And after two minutes was stopped due, to excessive vibration-. Inspection of the.turbine found no apparent problem; however, inspection of the outboard bearing housing found that there was no oil in the cavity,-although the sight glass showed
"'full.- The 'inboard.bearing showed 1/4 inch in the sight glas' and the cavity was found to have an.oil level 2 inches below theslinger ring. When the outboard bearing sight glass cap was opened, the sight glass was inspected and a piece of foam rubber ear plug was found in the sight glass'seat. Also observed at'that time was about half a quart of oil underneath the inboard bearing sight glass and none in the vicinity of the outboard'bearin Inspection of the bearing housing internals following the event revealed wear'.on the oil baffle (seal).' This condition.was a
,potential flow path for oil to migrate along the pump shaft and down to the turbine. base. Additional oil loss flow paths could have been through le king drain plugs and sight glasses. From the foregoing background information, the licensee developed the following scenario to account for the oil loss and. consequent wiping of the outboard and inboard turbine bearings of the steam driven AFW pump:
(1) August 6, 1985 -
Both bearing sumps were filled (a) Slow leakage noted from area of.both bearings over the perio (b) Inboard bearing' sump continually.kept full (c) Outboard bearing sump.continually and slowly lost oil, unreplenisied due to faulty indication caused by plugged sight glass (d) Approximately 14 hours1.62037e-4 days <br />0.00389 hours <br />2.314815e-5 weeks <br />5.327e-6 months <br />.of pump operation ensued (2) September 19, 1985.- Pump started automatically in response to plant trip (a) Outboard bearing failed due to loss of bearing oil (b) Vibration caused oil baffle (seal) failure of inboard bearing (c) Inboard bearing oil leaked out of damaged seal and bearing was destroyed (d) Excessive vibration caused mechanical trip of the pump
6, Specific Findings Involving Failure of the Unit 1 AFW Pump Based on NRC interview of involved station personnel, review of maintenance and QA/QC records, adreview of applicable station procedures, the following specific findings were.noted:
On February 22, 1985, MO #84112253-was completed on the Unit 1 turbine driven AFW pump. This MO performed the semi-annual preventive maintenance (PM) to change out the pump and turbine bearing lubricating oil. A copy of the completed MO recorde the following conditions with respect to bearingsight glasses:
"Outboard bearing sight glass on turbine was found 'Mickey Moused' to repair a leak... Vent on top of glass is frozen and cannot be vented properly. Millwriters handheld tubes to fill to proper level and then reinstalled brass outer housing. Hung deficiency tag 021259."
(2) The above referenced MO #84112253 was closed out by maintenance supervision on February 23, 1985, without initiating action to correct the above noted deficiencies. The MO was then forwarded to QA for final approval. Discussion with Controlled Document Management (CDM) personnel on October 3, 1985, identified that the MO work package was then in QA review for final sign off. Subsequent discussion with QA personnel on October 3, 1985, determined that the MO work package had not yet been transmitted to QA by maintenanc (3) A review of deficiency-tag files identified that tag #021259 (referred to in MO #84112253) was written against paint on a packing gland and not the-sight glas No deficiency written against the sight glass could be foun (4) A review of all MO's generated against the turbine driven AFW pump during 1985 identified that n6.MO was initiated to correct the deficiencies noted in MO #8411225 (5) A review of Work Authorization #850564 established that the turbine driven AFW pump was declared operable and returned to service on February 27, 1985, following completion of the preventive maintenance defined.in MO #8411225 (6) A review of the 3 deficiency tags still hanging on the steam driven AFW pump on September 20, 1985, identified that one of the tags (#11892, hung August 6, 1985) was written against a leaking AFW pump bearing sight glass and no corrective maintenance order had been~generated to correct the deficienc A random NRC review of 25 additional Unit 1 deficiency tags and 15 Unit 2 deficiency tags identified one other instance of a tag which was not traceable to an active MO. In this instance, a Unit 2 diesel generator.lube oil leak was noted by plant operators on September 1, 1985, and a corrective MO was not generated to-repair the leak until the discrepancy was noted by the-NRC inspector on September 30, 198 (7) Interviews.with maintenance personnel involved with the pump failure identified that:
(a) The individual whq improperly -installed the ear plug in the bearing sight.glass on-August 6, was a senior member of station maintenance supervisio (b) Prior to installing the ear plug, the above mentioned supervisor observed that the sight glass was not functioning and.was, in fact, plugged with a putty like substance. He corrected but did not document,or report this nonconforming condition on a safety'related componen (c) The maintenance craft person performing work on the pump on August 6, (MO #85052574) observed the supervisor making unauthorized repairs and failed to document or report this fact to station managemen (8) The above mentioned supervisory and craft personnel involved in the August 6 work (MO #85052574) had last received training in proper controls for maintenance work on safety related equipment in April 1984 and October 1984, respectivel (9) The individuals involved in the performance and approval of the February 22, 1985 work (MO #84112253) were Bechtel employees and had received training on the proper control of maintenance work on safety related equipment in March 198 (10) Nine MO's involving maintenance on the turbine driven AFW pump, generated since September 1984, were reviewed by the inspector Of those nine MO's, it was noted that five included only one review/approval signature. Although not clearly stated, it appears that the intent of procedure S0123-1-1.7 (MO Preparation and Use) is that.at least 2 different supervisors review and approve completed MO' (11) Observation of the performance of and interviews with station operations personnel responsible for maintaining proper bearing lubrication on safety related equipment identified the following:
(a) Operations personnel are not sufficiently sensitive.to the significance of bearing lubrication related deficiencie For example, the following deficiencies were noted by NRC inspectors:
An excessive quantity of lube oil was observed leaking from the motor bearings on Unit.2 AFW pum P141 on.August 6, 1985, without initiation of a deficiency tag or proper cleanup. Although reported to station maintenance and operations supervision at the time, the same conditions were again noted on September 20, 198 A leak of the magnitude observed
on the large capacity motor bearing oil system could easily mask leakage problems on the much smaller capacity pump bearings, in addition to being a poor work and housekeeping practic Out-of-specification high oil level was observed on the outboard pump bearing of Unit 2 AFW pump 2P140 on August 6, 1985, without initiation of a deficiency tag. This condition still existed when the inspector observed the pump on September 20, 198 o A recognized out-of-specification high level existed on the motor bearing oil cooling system drain tank of Unit 2 AFW pump 2P141 from August 17,.1985 until it was drained on October 2, 1985 after being questioned by the NRC inspecto o Excessive oil leakage was observed on October 10, 1985 on Unit 1 and Unit 2 charging pumps, without initiation of a deficiency ta (b) Operations personnel were making at least daily inspections of'all safety related pumps with the following exceptions:
o The Unit 1 recirculation pumps are located in
..containment and have sealed beating o The Unit 1 positive displacement charging pump has no visible oil monitoring indicator; maintenance personnel perform a semiannual check of oil level by use of a dip stic (c) As noted above with the:Unit 1 charging pump, some bearing oil systems haveno sight glass or other 'ethod to directly observe bearing oil leve On these types of bearing lubrication systems the.'proper functioning of automatic oilers (if installed) takes n additional importance and possible design change or additional training emphasis may be warranted (206/85-33-0.1).
(12) Observation of the performance of and 'interviews with station maintenance personnel responsible for ridpair and,iaintinance of safety related equipment identified that.these personnel were knowledgeable with 'regard to procedure requirements related.to observed nonconforming conditions and"the performance 'of work not covered in a safety related 0."
(13) An enforcement conference with the licensee was held in Region V on March 25, 1985, dealing with maintenance related problems associated with improper functioning of a Unit 1 containment air lock. One aspect of this conference dealt with the resource that is available within the licensee organization in the form of the cognizant plant equipment engineer. During
this conference the licensee emphasized their intent to'take greater advantage of this resource to provide additional, ( knowledgeable overview of operational and maintenance activities associated with important safety related system During discussions with the NRC inspector, the Site Manager stated that additional emphasis in this area of overview appears to be warranted in order. to help prevent some of the operational and maintenance deficiencies noted in this repor (14) Considering the circumstances involved with the failure of the Unit 1 AFW pump and the findings noted above, the Site Manager stated that a comprehensive review of the subject of bearing lubrication for safety related pumps is being conducted by the licensee. This review will address the design, operation and surveillance of bearing lubrication systems and will assess the need for additional training on these system Conclusions Resulting from NRC Review (1).As a result of failure to comply with station administrative procedures for controlling work on safety related equipment, the steam driven AFW pump failed-during Mode 3 operation when it was required by Technical Specifications to be operabl The following station procedure violations were noted (206/85-33-02):
(a) No deficiency tag or MO was generated to correct observed nonconforming conditions on the AFW pump sight glass observed on February 22, 1985 (MO #84112253), as required by station procedure SQ1-14-1 (b) Maintenanrce supervision did not perform an adequate review of completed MO #84112253, as required.by station procedure SO1 I-1.4.0 (SQ123'I' by a
.(c)
Maintenance personnel did not repot r document nonconforming conditiohs on the At opump sght glass when again observed on August 6, 1985, as required by station procedire S0123-XV- (d) Deficiency tags were.written agaiist problems affecting safety relatedequipment and no MO{s were generated to track or correct the defiieficies, in Violation of station
.
procedure S01-14-1 Maintenance personnel improperly modified the Unit 1 AFW pump sight glass on August 6, 1985o, in apparent violation.of Technical Specification 6.5.1.11 and Station Procedure S01-1-1.4.0, which requires that any modifications to safety related equipment be designed bya qualified individual/organization with independent review by another individual/organization (206/85-33-03).
(3) The violations addressed'.in paragraphs (1)(c) and (2) above were committed by a seniqr member of station maintenance supervision. Furthermore, these violations were observed by station maintenance craft personnel and'not reported to station managemen (4) Completed MO packages were not beitig.submitted to QA for final review in a timely manner as intended by station procedure 30123-1-1.7 (MO Preparatiod, Use & Scheduling).' Quality Assurance representatives stated that procedure changes would be made to ensure more timely submittal for QA revie. Exit Meeting On October 11,1985, an exit meeting was conducted with the licensee representatives identified in paragraph 1. The inspectors summarized the inspection scope and findings as described in 'this report.