IR 05000275/1989013
| ML16341F142 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 04/18/1989 |
| From: | Johnston K, Mendonca M, Narbut P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML16341F141 | List: |
| References | |
| 50-275-89-13, 50-323-89-13, NUDOCS 8905110315 | |
| Download: ML16341F142 (40) | |
Text
U.
S.
NUCLEAR REGULATORY COMMISSION
REGION V
Report Nos:
50-275/89-13 and 50-323/89-13 Docket Nos:
50-275 arid 50-323 License Nos:
Pacific Gas and Electric Company 77 Beale Street, Room 1451 San Francisco, California 94106 Facility Name:
Diablo Canyon Units 1 and
Inspection at:
Diablo Canyon Site, San Luis Obispo County, California Approved by:
Inspection Conducted:
January 22 through March 17, 1989 and April 6, 1989
,~.~aa Mg~a l~P K.
E. Johnston, Resident Inspector Date Signed egzs/s p P.
P.
Narbut, Senior Resident Inspector Date Signed C'
M.
M. Mendonca, Chief, Reactor Projects Section
Date Signed Summary:
Ins ection from Januar 22 throu h March 17 1989 and A ril 6 1989 Re ort Nos.
50-275/89-13 and 50-323/89-13 Areas Ins ected:
This special inspection was conducted to examine the circumstances surrounding several occurrences of inoperability of the auxiliary feedwater pumps experienced in the period from December 1988 to February 1989.
The inspection was conducted by the resident inspectors and the majority of inspection hours were charged to their regular monthly inspection report (50-275/89-05 and 50-323/89-05) prior to the decision to issue this special report.
This report also includes an examination of the Licensee Event Reports (LERs) written in response to the major team inspection findings reported in inspection reports 50-275/89"Ol and 50-'323/89-01.
A meeting with the licensee was held on April 6, 1989 to discuss the technical details of.the LER's.
Results of Ins ection:
The inspection resulted in the identification of three apparent violations which will be the subject of separate correspondence.
This report provides the facts surrounding the apparent violations.
The apparent violations involve an inoperable auxiliary feedwater (AFW) pump due to an open valve which caused a harsh steam environment in the pump room, an inoperable AFW pump due to an isolated steam supply, and an inoperable overspeed protective device for the AFW pump steam turbine due to an inadequate maintenance procedure for lubrication and testing of the overspeed trip device.
8905ii03i5 8904i9 PDR ADOCK 05000275 Q
DETAILS 1.
Persons Contacted
- J.
D. Shiffer, Vice President, Nuclear Power Generation J.
D. Townsend, Plant Manager D.
B.
Miklush, Assistant Plant Manager, Maintenance Services L.
F.
Womack, Assistant Plant Manager, Operations Services
¹"B.
W. Giffin, Assistant Plant Manager, Technical Services
"J.
M. Gisclon, Assistant Plant Manager for Support Services C.
L. Eldr idge, equality Control Manager T.
A. Bennett, Maintenance Manager
- D. A. Taggert, Director guality Support W.
G. Crockett, Instrumentation and Control Maintenance Manager J.
V. Boots, Chemistry and Radiation Protection Manager
¹"T.
L. Grebel, Regulatory Compliance Supervisor
"M. J.
Angus, Work Planning Manager J.
A. Shoulders, Onsite Project Engineering Group Manager
"M.
E.
Leppke, Engineering Manager
"S.
R. Fridley, Operations Manager R.
P.
Powers, Radiation Protection Manager
"K. Doss, Onsite Review Group
"W. J. Kelly, Regulatory Compliance Engineer
- K. J.
Condron, Assistant Project Engineer
¹ M.
R. Tresler, Project Engineer
¹ S. Auer, Electrical Engineering
¹ R.
B. Clark, Mechanical Engineering
¹ J.
L. Kelly, Mechanical Engineering
¹ D. J.
Hampshire, Nuclear Regulatory Affairs
¹ B.
S.
Lew, Nuclear Regulatory Affairs The inspectors interviewed several other licensee employees including shift foremen (SFM), reactor and auxiliary operators, maintenance personnel, plant technicians and engineers, quality assurance personnel and general construction/startup personnel.
Denotes those attending the exit interview on March 17, 1989.
Denotes those attending the meeting in Walnut Creek on April 6, 1989 also attended by Mssrs D.
F. Kirsch, F.
R.
Huey, A.
E. Chaffee, M.
M. Mendonca and S.
A. Richards (by telephone)
of the NRC Region V offices.
2.
General Overview During the period from December 1988 to February 1989 a number of incidents revealed the fact that the auxiliary feedwater system, particularly the Unit 2 steam driven auxiliary feedwater pump had been inoperable for period of time.
This report describes the circumstances of the three occurrences of auxiliary feedwater component unavailabilit The examination of these occurrences also led to corollary issues as described in the resident inspector monthly inspection report (50-275/89-05 and 50-323/89-05).
Specifically, these corollary issues involved weaknesses in the licensee's recovery actions from the discovery of an inoperable overspeed trip device for the steam driven AFW pump turbine.
The weaknesses described included improperly authorized parts modifications (effectively a design change)
and poor configuration control (in that an inappropriate drawing was supplied for craft use).
These corollary issues wi 11 not be detailed further in this report.
The three occurrences described in this report indicate that the root causes of the inoperabilities are not new, and have been the subject of previous inspection reports and/or management meetings.
The inoperable AFW pump, due to an open valve which created a harsh steam environment, was caused by poor valve lineup controls which has been a
noted weakness in the last several resident inspector reports and has been the subject of licensee event reports for other inoperable equipment (containment cavity sump level indication lost due to a valve lineup error) and reactor trips (a reactor trip due to an anti-motoring relay, pressure sensing line that was inadvertently valved out).
The failure to declare an AFW pump inoperable when one steam supply valve was isolated was the result of a lack of understanding of design basis information by plant personnel.
This lack of understanding of design basis information has been the subject of several management meetings and the subject of violations in the last two team inspections in July 1988 and February 1989.
The inoperable overspeed protection device due to rust accumulation was caused by a combination of two facts:
First, longstanding steam leaks into the room had not been corrected, and secondly necessary periodic maintenance and testing had not been implemented.
Timeliness of addressing and correcting problems was an issue addressed in the 1988 SALP report.
The need to evaluate and implement necessary maintenance and testing for components, and the need to assure proper engineering overview and design bases maintenance were major subjects of the maintenance team findings identified in July 1988.
Auxiliar Feedwater Pum Problems The following sections describe three separate auxiliary feedwater pump unavailabilities experienced from December 1988 to February 1989.
a.
Ino erable Auxiliar Feedwater Pum Due to an Im ro erl 0 en Vent Valve from December 1 to December
1988 In inspection report 50-323/88-30, an unresolved item (88-30-01)
dealt with the Unit 2 steam driven Auxiliary Feedwater Pump 2"1, which was discovered in an inoperable state when personnel performing a monthly surveillance test were driven out of the pump room when steam was admitted to the pump and blew into the room through an open instrument root valve and its uncapped pipe nippl The pump had been previously successfully tested on November 30, 1988, at 5:11 a.m.
PST.
The pump was taken out of service for a oil change on November 30, 1988, at 1:49 p.m.
PST and returned to service on December 1, 1988, at 7:00 a.m.
PST.
The post maintenance verification consisted of an oil level check and a return to service valve lineup.
The pump was again attempted to be tested on December 31, 1988, at 1:29 a.m. (for its normal monthly operational test).
At this time the open instrument root valve and missing pipe nipple cap became evident as personnel were driven out of the pump room.
The valve (MS-2-923) is ordinarily opened (and the pipe cap removed)
as part of the clearance for maintenance, such as the oil change, to provide local assurance that steam is in fact removed from the turbine.
The valve, MS-2-923, and cap were to have been restored to normal following the November 30, 1988, oil change and the valve was documented as closed and verified closed by two auxiliary operators at 5:33 a.m.
on December 1,
1988.
The clearance valve lineup did not specifically address the cap installation.
Licensee management interviews with the two auxiliary operators were stated to be not totally conclusive in that the personnel did not remember details of the specific valve lineup over a month after it was performed but were certain that they had performed it properly to the best of their recollection.
Other avenues of investigation such as reviewing the possibility of other maintenance work orders, I8C work or engineering tests did not provide any indication of other work performed.
The licensee prepared and issued Licensee Event Report (LER) 88-24 on January 30, 1989, which postulates that either the two auxiliary operators left the valve mispositioned and the line cap unplugged or the valve and cap were subsequently mispositioned.
It is reasonable to assume that the valve and cap were mispositioned from the point in time immediately after they were last verified, i.e., after the oil change on December 1, 1988, at 5:33 a.m.
PST.
The licensee LER also states that an engineering review concluded that the steam driven pump would have failed due to high room temperature if required to operate for an extended period of time.
The LER also states that the licensee's review of operating records indicate that the motor driven auxiliary feedwater pumps were operable during this time period.
The Technical Specifications for Diablo Canyon, section 3.7. 1.2, require that with one auxiliary feedwater pump inoperable for 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> the unit be put in hot standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />.
Contrary to the above, Unit 2 operated from December 1, 1988, at 7:00 a.m.
PST to December 31, 1988, at 1:29 a.m.
PST, a total of 714 hours0.00826 days <br />0.198 hours <br />0.00118 weeks <br />2.71677e-4 months <br /> and 29 minutes with an inoperable auxiliary feedwater pum The pump was inoperable due to the open steam line vent valve MS 2-923 which would have created a damaging steam environment in the pump room, should a actuation of the pump been required.
This is an apparent violation of Technical Specifications (Item 50-323/89-13-01).
Enforcement will be the subject of separate correspondence.
The licensee's corrective action as described in LER 2-88-24 and nonconformance report NCR DC 2-89-0P-N005, reflect that a root cause could not be positively determined and corrective action to prevent recurrence is being investigated and considered.
The resident inspectors previous monthly reports have emphasized valve lineup problems as an area of weakness.
Inspection report 50-275/89-06 describes the NRC concern regarding valve lineup errors expressed by the Deputy Regional Administrator to plant management on January 26, 1989.
Inspection report 50-275/89-05 describes a feedwater transient caused by the injection of air into the feedwater system, in turn caused by an improper valve lineup.
Also identified was a lack of control of pipe caps and plugs in valve line up procedures.
Inspection report 50-275/88-32 discusses the perceived lack of action on the part of the licensee in dealing with valve lineup and configuration control problems.
Inspection report 50-275/88-31 describes missing seals on valves on the auxiliary feed water system which required testing to verify proper positioning, unplanned closure of a safety injection pump suction valve leading to damage of the pump and ineffective quality assurance surveillances of the valve line up process.
Inspection report 50-275/88-30 describes the SALP management meeting held in Walnut Creek on October 26, 1988 in which some of the specific topics discussed related directly to the area of valve lineup problems.
Specifically the licensee was informed of and recognized that they had been slow to implement lessons learned from operating experience and that correct instincts had not been developed to follow procedures or stop in the face of uncertainty.
Inspection report 50-275/88-26 described valve lineup errors leading to overfilling a steam generator, the injection of water into the containment nitrogen system, and erroneous level indications in the reactor vessel level indicating system for refueling.
Additionally the report described a lack of adequate valves seals on CCW valves.
This report was forwarded to licensee management with a cover letter requesting that the licensee describe their plans for dealing with the underlying causes of the events of overfilling the steam generators and injecting water into the nitrogen system (and other events).
The licensee responsed on January 6,
1989 reporting on their progress in instilling proper instincts in personne Additionally, the licensee has incurred previous reportable events caused by valve line up errors.
Specifically, LER 1-88-026 dealing with a Unit 1 reactor trip caused by a mispositioned root valve for an turbine anti-motoring relay, LER 2-88-13 dealing with an inoperable reactor cavity sump level device from April 1987 to September 1988 caused by a mispositioned instrument air valve, and LER 1-88-30 dealing with an inoperable containment radiation monitor due to an improper valve alignment.
Consequently, it is judged that the licensee's corrective actions for valve lineup problems have been neither thorough or timely.
Personnel Interviews The inspector noted that the incident of blowing steam in the steam driven auxiliary feedwater pump room on December 31, 1988, had not been noted in either the auxiliary operator's log, the control operator's log, or the shift foreman's log.
The inspector questioned the individuals involved and concluded that the individuals had not considered the occurrence logable.
However, the incident was reported in the shift supervisors electronic turnover notes.
Additionally, the auxiliary operator wrote an action request (A/R) on the incident on his following 12 hour1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> shift.
The inspector performed the oil change on and verifier, best of their interviewed the two auxiliary operators who had last known valve manipulation of MS 2-923 after the November 30, 1988.
Both individuals, the positioner attested to proper positioning of the valve to the recollection.
Procedure and Drawin Deficiencies The inspector noted that the post outage valve lineup of the auxiliary feedwater system had been conducted in accordance with operating procedure OP D-12: II, Revision 5 between October 27 and November 26, 1988.
The procedure entitled "Operating Procedure-Auxiliary Feedwater System Alignment Verification for Plant Startup" lists valve MS 2-923, but does not address the pipe nipple cap.
Likewise the operating valve identification drawing, OVID, PG5E Drawing 107704 Revision 12 shows valve MS 2-923 closed but further shows it as in-service to isolate pressure gage PX-213 which has" been removed.
The drawing does not represent the actual plant configuration which has the gage removed and the pipe nipple capped and plugged.
The inspector discussed the valve lineup procedure and drawing deficiencies with the operations manager who stated that revisions to valve lineup procedures and OVID were planned as part of the actions being identified in the nonconformance report written in response to valve lineup errors in general.
The general time frame for these procedure and drawing improvements was stated to be two years in accordance with the licensee's standard two year review schedul After a system is initially lined up for operation in accordance operating procedures such as OP 0-12: II, changes are controlled, through the clearance process as defined by the administrative procedure AP C-6S1 Revision 15 "Clearance Request/Job Assignment".
The clearance process provides for the realignment of valves temporarily to perform testing or maintenance and provides for the return of the valves to normal operating position after the test or maintenance is completed.
The clearance process is the method the licensee utilized to maintain proper system alignments after the initial lineup.
The inspector noted that the clearance valve lineup used to perform the oil change on December 1, 1988, properly positioned valve MS 2-923 but did not address the installation of the pipe nipple cap.
Operators stated that although most clearances do not address the presence or absence of such caps, the operators check for the presence of caps as a matter of course.
The inspector discussed the apparent deficiency in the clearance valve lineup process (not specifying caps) with the work planning manager.
The work planning manager stated that he would revise instructions to work planners to include the removal and reinstallation of caps in clearance valve lineups.
As a footnote, on March 1, 1989, the licensee discovered the Unit 1 turbine driven AFW pump had the pipe cap plug missing from the pipe nipple attached to valve MS 1-923.
The valve itself was closed, however.
Safet Si nificance The auxiliary feedwater system is designed with three independent feedwater pumps, one steam driven and two motor driven.
The technical specification bases state that the operability of the system ensures that the reactor coolant system can be cooled down to less than 350 degrees F in the event of a total loss of offsite power.
The Diablo Canyon FSAR, Section 6.5. 1. 1, further states that the auxiliary feedwater system is also designed to provide protection against the loss of all a.c.
power.
Section 6.5. 1. 1.3 explains that in the event of the loss of all onsite and offsite a.c.
power, decay heat would be removed through the availability of the one steam driven auxiliary feedwater pump.
However, Chapter 15 of the FSAR does not include loss of all a.c.
power as a required accident for analysis.
The inspector therefore concluded that for the studied accidents of Chapter 15 of the FSAR, the AFW system was capable of functioning as required by virtue of the availability of the two motor driven auxiliary feedwater pumps.
In the event of a total loss of a.c.
power, however, the design feature of being able to remove decay heat through the steam generators would not have been availabl Conclusion The unavailability of the steam 'driven auxiliary feedwater pump 2-1 for a period 'of about 30 days in December 1988, represented a
serious loss of capability to deal with a loss of all a.c.
power event.
Unresolved item 50-323/88-30-01 is considered closed and superseded by apparent violation 50-323/89-13-01 previously described in this report.
Unit 2 Turbine Driven Auxiliar Feedwater Pum Ino erable due to One Steam Su
Out Of Service Januar 17 to Januar
1989 Technical Specification 3.7. 1.2 requires that three steam generator auxiliary feedwater pumps and associated flow paths shall be OPERABLE with the one steam turbine-driven auxiliary feedwater pump capable of being supplied from an OPERABLE steam supply system.
On January 17, the main steam line 2-2 steam supply valve to turbine driven AFW pump 2-1, FCV-2-37, was closed with power removed without declaring AFW pump 2-1 inoperable while the plant was in power operations (Mode 1).
Concurrently, a motor driven AFW pump was declared inoperable with it's discharge isolation valve (FW-2-190)
shut for maintenance of LCV-2-115, AFW supply to steam generator 2-3.
Following questions by the resident inspector, the licensee on February 3, 1989, determined that, with FCV-2-37 closed, the AFW turbine steam supply system would not have performed its design function as described in Chapter 15 of the licensee's Final Safety Analysis Report (FSAR) under certain conditions.
The licensee further determined that AFW pump 2-1 should have been declared inoperable in accordance with technical specifications for a limited time.
Summar S stem Descri tion Each Unit has three auxiliary feedwater pumps, two motor driven and one turbine driven.
Each motor driven pump (AFM pumps 2-2 and 2-3)
feeds two steam generators and can be cross-tied at the pump discharge.
The turbine driven pump (AFW pump 2-1) supplies all four steam generators.
The turbine has two steam supplies from main steam lines two and three.
Each line has a motor operated isolation valve, FCV-37 and FCV-38 from main steam lines two and three,
"respectively.
The two supply a header upstream of FCV-95, the normally closed D.C. motor operated valve which opens on a start signal.
Chronolo of Events On January 17, 1989, at 5: 12 a.m.,
motor driven AFW pump 2-3 was declared inoperable, and the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement for technical specification 3.7. 1.2 was entered for maintenance on 2-LCV-115.
At 5: 27 a. m.,
AFW pump 2-3 discharge valve, FM-2-190, was closed in accordance with the same clearance, C/R 1725 In addition, for a separate maintenance activity, at 5: 13 a.m.,
FCV-2-37 was closed and power removed in accordance with C/R 17243.
However, AFW pump 2-1 was not declared inoperable.
On January 18, 1989, at 6:30 a.m.,
AFW pump 2-3 was returned to service and the action statement was exited.
On January 19, at 1:36 a.m.
power was restored to FCV-2-37 and the valve was declared operable at 6:30 a.m.
On January 20, in a review of the January 17 Shift Foreman logs, the resident inspector noted the above and on January 25 brought it to the attention of the Operations Supervisor, requesting confirmation that FCV-2-37 and AFW pump 2-3 were out of service concurrently and requesting confirmation that with FCV-2-37 out of service, AFW pump 2-1 should have been declared inoperable.
Subsequent discussions were held between the resident inspectors, the Diablo Canyon Project Engineer, the Assistant Plant Manager for Operations Services, and the Assistant Plant Manager for Technical Services.
In these discussions it was determined that with FCV-2-37 out of service, AFW pump 2-1 was inoperable because its steam supply system could not have performed a design function as specified in Chapter 15 of the FSAR under certain conditions.
Safet Si nificance The design requirements for the AFW system are in the FSAR.
Section 15.4.2.2 of the FSAR, "Major Rupture of a Feedwater Pipe," states that following a postulated feedline break between the last check valve and a steam generator;
"The AFW is assumed to be initiated 10 minutes after the trip with the feed rate of 440 gpm.
An additional 4 minutes is assumed before the relatively cold (120 degree F)
AFW enters two of the three unaffected steam generators."
The design of the AFW systems assumes a single fai,lure.
This design requirement is not met if one of the steam supplies is closed assuming a single active failure and the rupture of a specific feedwater pipe.
Specifically, if one supply is taken out of service (as an example; FCV-2-37),
and a feedline break is postulated on the feedline of the redundant steam supply (Steam Generator (S/G) 2-3); then AFW pump 2-1 is not available because it has. no steam supply.
Further, if the single active failure is postulated to fail the motor driven AFW pump which supplies two good steam generators (in this case AFW pump 2-2 which supplies S/Gs 2-1 and 2-2), the remaining AFW pump (AFW pump 2-3) would supply the faulted S/G and one operable S/G (which does not meet the Chapter
FSAR design requirement previously described to feed two unaffected steam generators).
It should be noted that the Limiting Conditions for Operation (LCOs)
in the TS are based on the assumptions made in the FSAR accident analyses.
This provides assurance that when the LCOs are met, single failure protection is provided, because the accident analyses
assumes a single active failure.
Entry into an action statement occurs because of some inoperability.
Under these conditions, functional capability still exists, but single active failure protection does not.
For this reason, the action statement allows continued operation for some limited period of time.
Therefore, when an action statement has been entered, it is no longer appropriate to postulate a single active failure when evaluating the analyzed accident scenarios upon which the TS is based.
Based on the above, with either FCV-2-37 or FCV-2-38 inoperable, AFW pump 2-1 is inoperable and appropriate declaration and action statement should be entered.
Between January 17 and 19, the plant was in two distinct conditions:
In the first condition (January 17, 5: 13 a.m. to January 18, 6:30 a.m.), given the TS interpretation described above, both AFW pump 2-1 and AFW pump 2-3 were inoperable and the Action Statement for two AFW pumps inoperable requires achieving Hot Shutdown within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> which was not complied with.
However, in this condition, since the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement for one AFW pump inoperable was entered, as discussed above, it is not appropriate to assume a single active failure.
Therefore, the design basis was met since unless a single failure is assumed, AFW pump 2-2 was available to supply two steam generators.
In the second condition (January 18, 6:30 a.m. to January 19, 6:30 a.m.), after AFW pump 2-3 was returned to service, AFW pump 2-1 was still inoperable with FCV-2-37 closed and no action statement had been entered.
In this configuration, absent the formal declaration of inoperability, and the entry into the TS 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement, the licensee is not in compliance with its design basis since AFW pump 2-2 can be assumed to fail concurrent with a line break on SG 2-3 (resulting in no steam available to AFW pump 2-1).
However, although the action statement was not formally entered, the licensee did not exceed the action time of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> which would have been allowed if a formal declaration of inoperability had been made (FCV-2-37 was returned to service 49 hours5.671296e-4 days <br />0.0136 hours <br />8.101852e-5 weeks <br />1.86445e-5 months <br /> and
minutes after it had been removed from service).
Therefore, contrary to the actual statement b. of Technical Specification 3. 7. 1. 2, from January 17, 1989, at 5: 13 a.m. to January 18, 1989, at 6:30 a.m., with two Unit 2 auxiliary feedwater pumps inoperable, the licensee did not proceed to HOT STANDBY within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and HOT SHUTDOWN within the following 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />.
Auxiliary feedwater pump 2-3 was declared inoperable from January 27, 5: 12 a.m. to January 18, at 6:30 a.m. with its discharge valve closed for maintenance on steam generator level control valve LCV-2-115.
Auxiliary feedwater pump 2-1 was inoperable, but not declared inoperable, from January 17, 1989, at 5:13 a.m. to January 19, at 6:30 a.m., with FCV-2-37 (the turbine steam supply from main steam line 2-2 isolation valve) closed with power removed for maintenance on the valve motor operator.
The licensee had not recognized that
with FCV-2-37 closed with power removed, that the auxiliary feedwater turbine steam supply system was inoperable.
This is an apparent violation (Item 50-323/89-13-02).
Enforcement will be the subject of separate correspondence.
The technical significance of the proposed violation is reduced for the following reasons:
1)
The probability of the sequence of events required for loss of AFW to three S/Gs is very low and even with worst case assumptions, the licensee's emergency procedures contained appropriate steps to identify and mitigate the consequences.
2)
As explained above, from January 17 at 5: 13 a.m. to January
at 6:30 a.m., with two pumps inoperable, although not in compliance with the technical specification 6 hour6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> action statement, the licensee was not outside its FSAR design basis.
Additionally, from January 18 at 6:30 a.m. to January 19 at 6:30 a.m., with one pump inoperable (but not declared so), the licensee was outside its design basis under certain narrow conditions; about a full day existed before the licensee would have exceeded the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement had they recognized and formally declared AFW pump 2-1 inoperable.
However, it should also be noted that for a number of other reasons, the significance of this violation is enhanced.
First, operators and the clearance coordinator had the opportunity to recognize that with one AFW pump out of service, removing from service a valve related to the power supply of a redundant pump reduces the overall redundancy of the AFW system to respond to accident conditions, and is, therefore, a less conservative configuration.
Second, guidance provided in the available licensee procedures was ambiguous and the decision based on the procedures was not a conservative one.
Finally, and most significantly, this is an example of the plant staff's lack of understanding of plant design basis and it'
thorough implementation into plant procedures.
This issue was previously identified by the Maintenance Team Inspection (Inspection Report 50-275/88-15)
and the Safety System Functional Inspection (Inspection Report 50-275/89-01)
and it further indicates that continued efforts are needed to understand and implement the design bases.
In this case the licensee engineering organization had clear design basis information available in the form of the Westinghouse Steam Systems Design Manual Revision 2, dated August 1973 (WCAP-7451).
The manual specifically stated that for the auxiliary feedwater system turbine driver, steam must always be available from at least two steam generators during plant operation to preclude a loss of all steam supplies due to any single incident.
Root Cause The licensee initiated a non-conformance report for this event.
The root cause was identified to be that no definitive guidance was
provided which identified that an operable steam supply system required two operable steam supplies.
It is the operations department's policy that for situations where specific equipment is not called out by the Technical Specifications that operations personnel refer to the surveillance test procedures (STPs),
to determine the component's effect on system requirements, and operability.
The inspector interviewed both the clearance coordinator and the shift supervisor who authorized the clearance and found that they had complied with this guidance and reviewed all related survei llances.
The inspector reviewed the STPs and found that none specifically required that both steam inlet valves be operable.
Further, the licensee issued a Licensee Event Report on this incident (LER 50-323/89-01, dated February 24, 1989).
The licensee determined the root cause to be that design information included in WCAP 7451 had not been incorporated into plant documents.
As a
.result, the clearance coordinator and the shift foreman did not have adequate guidance to determine conclusively that taking FCV"2-37 out of service made AFW pump 2-1 inoperable.
The inspector reviewed plant procedures and interviewed the clearance coordinator and concurred that existing procedures did not provide adequate criteria.
However, as stated above, the inspector found that the decision based on ambiguous procedures was not a conservative one.
A review of the licensee's corrective actions will be conducted in conjunction with a review of the LER and the licensee's response to the violation.
Ino erable Auxiliar Feedwater Pum Overs eed Tri Actuatin Device and Overs eed Tri Sto Valve FCV 152 Februar
1989 On February 12, 1989, at 10:45 p.m.
PST, the licensee discovered that the overspeed trip device for the turbine driven auxiliary feedwater (AFW) pump 2-1 was inoperable.
The pump was declared inoperable and a 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action statement was entered in accordance with Technical Specifications.
The discovery of the inoperable overspeed device was made inadvertently during training of an auxiliary operator.
The trainee was being taught how to trip and relatch the device.
Upon manual tripping, the overspeed device did not cause the overspeed stop valve (FCV 152) to go shut.
This was a cold test, steam had not been admitted to the line.
Subsequent maintenance investigation revealed two independent problems'irst, the valve FCV 152, a spring-actuated to-close-valve, did not close even after the latch-open mechanism was physically tripped open.
The problem was subsequently determined to be heavy rust and corrosion on the spring-to-close mechanism which inhibited movement of the valve stem assembly.
Secondly,= the trip mechanism itself did not move sufficiently to unlatch the valve.
This was due to corrosion and hardened lubricant on some parts of the assembl The licensee devised an action plan to investigate and correct the situation.
These actions included:
o The inspection of Unit 1 for similar conditions (Unit 1 was inspected and tested and found to be satisfactory).
o Modified testing requirements to increase the testing frequency from every refueling outage to monthly.
o Perform a review of the overspeed trip mechanism for design acceptability.
o Assess drawing adequacy for the mechanism.
o Initiate an engineering request to provide drainage from the FCV 152 spring assembly.
o Prepare detailed trip mechanism maintenance procedures.
The licensee cleaned and repaired the valve, actuation assembly and trip mechanism.
The mechanism was satisfactorily tested and declared operable on February 14, 1989.
The licensee had preliminarily concluded that the condition was caused by an excessive steam environment in the AFW pump room due to previous problems with steam trap drains and current problems with building heating steam leakage paths.
The licensee had been dealing with steam leaks in the pump room since 1985 per the operations manager.
Although certain sources had been resolved, excessive steam environmental conditions continued to be an unresolved problem.
The resident inspectors examined the situation and have preliminarily determined that three main problems appear to be involved.
First, the licensee had not implemented actions recommended in the vendor technical manual specifically to test the overspeed trip device monthly and to lubricate the device weekly.
Secondly, the licensee's corrective work orders lacked specificity and provided mechanics with drawings of a different (later) model of overspeed trip device.
Thirdly, the licensee had previously experienced problems with the overspeed trip device in November 1988, during testing after the Unit 2 second refueling outage.
This previous occurrence identified the lack of proper drawings for the device but the work order line item to obtain proper drawings had been "N/A'd" by a site engineer.
These matters were brought to the licensee management's attention.
Safet Si nificance The steam driven auxiliary feedwater pump is regulated in speed by a Woodward governor and valve.
The vendor technical manual (PG8E document Number DC 663056 Revision 8) contains several warning notes that the turbine device should be protected with a separate overspeed shutdown device should the governor fai Therefore, the licensee's operators declared the pump inoperable upon discovery of the overspeed device problems.
Licensee engineering personnel concluded that the pump would have performed its design function if the governor functioned properly (which it did during functional testing following the event)
~ It remains also true that had the governor not functioned properly the lack of overspeed protection could have resulted in probable AFW turbine/pump damage and feedwater piping over presurrization.
Period of Ino erabilit The exact period of inoperability of the overspeed trip device could not be determined since the buildup of rust would be a gradual process.
,The licensee had performed overspeed trip testing on a
refueling outage frequency.
The history of testing was:
A o
July 21, 1985, test satisfactory o
June 1987, final test satisfactory, slight trip speed adjustments were required o
November 25, 1988, final test satisfactory, initial test failed to tr ip, maintenance and spring replacement performed.
Additional Weaknesses Additional weaknesses were identified by the resident inspectors such as the corrective maintenance work order (WO 49505),
issued to investigate and repair FCV 152, was lacking in detail and provided the craft with a drawing of an overspeed trip device different than that installed.
Additionally, the resident inspectors found during the recovery work that improperly authorized parts modification (a washer)
was performed and poor configuration control was exercised.
These subjects are discussed in the resident inspection report 50-275/89-05 and.50-323/89-05.
Conclusion The absence of a licensee program to perform vendor recommended mai,ntenance and testing of the overspeed trip device, FCV 152, is considered a violation of NRC requirements to perform safety related work in accordance with procedures appropriate to the circumstances (Item 50-323/89-13-03).
4.
S ecial Licensee Event Re ort Evaluation Subsequent to the NRC team inspection reported in inspection report 50-275/89-01, the licensee submitted Licensee Event Reports (LER's)
dealing with the major findings of the team inspection.
The inspection report stated that the LER's would be reviewed.
On April 6, 1988 a
meeting was held in the Region offices in Walnut Creek with licensee engineering representatives, the resident inspectors and regional management personnel to discuss the LER's and their technical conten Com onent Coolin Water CCW and Auxiliar Saltwater S stem ASW Desi n Basis Re uirements Not Incor orated Into Plant Procedures
~/
An NRC Safety System Functional Inspection Overview Team in January 1989 (see Inspection Report 50-275/89-01) identified these problems.
On March 24, 1989, the licensee issued Licensee Event Report (LER)
50-275/84-40,
"CCW and ASW System Design Basis Requirements Not Incorporated Into Plant Procedures Due to Inadequate Tracking of Resolution of Correspondence and Communications."
The report was made following the licensee determination, on February 22, 1989, that engineering requirements for plant operation to assure compliance with the design basis were not incorporated in plant operating procedures and emergency procedures.
The Engineering requirements were included in January 3, 1984, and February 14, 1984, letters to the plant.
~Anal sis:
The analysis provided in the LER bases its determination that there was no compromise to the safe operation of the plant on two revised calculations.
The first shows acceptable operation with one ASW pump and one CCW heat exchanger under certain conditions.
The second shows acceptable operation with one CCW pump supplying all three CCW headers.
The review of the acceptability of these analysis was discussed in an April 6 meeting with the licensee and it was agreed that the new calculations indicate that the ultimate effect of not implementing engineering's 1983 recommendations for operation would not have apparently caused unsafe operation.
It was further agreed that the new analyses would require submittal and NRC review if the licensee now chooses to implement the results in plant operation.
Root Cause:
The licensee's determination that the root cause was
"inadequate tracking of resolution of correspondence and communications specific to engineering design basis constraints on plant operations" was found to be acceptable.
Corrective Actions:
In addition to revising procedures to reflect the requirement to place a second heat exchanger in service when CCW pumps l-l and 1-2 are out of service, the licensee committed to take broader action to perform an expeditious review of the FSAR and their correspondence files for similar design requirements, to include procedure reviews in the Configuration Management Program (CMP) and to revise their design change procedure to specify that Engineering identified operating constraints be communicated by the design change process.
Im ro er Desi n Chan e Packa e for Auxiliar Saltwater S stem Pum Im eller Re lacement LER 50-275/88-32 On March 29, 1989, the licensee issued LER 50-275/88-32 regarding the improper design change package for the ASW pump impeller replacement.
This report resulted from two related findings by the NRC team inspection, conducted in January, 1989.
The first
identified that a motor overcurrent setpoint had not been adjusted to compensate for the greater horsepower needed to drive the new impeller.
Although the licensee had performed a calculation to verify the adequacy of the overcurrent setpoint, the calculation did not consider the design requirement of maintaining pump operation with a 90K undervoltage condition.
The second design weakness resulted from the failure to consider the impact of increased motor horsepower on emergency diesel generator fuel consumption and the consequent minimum fuel storage requirements.
~Anal sis:
ASW Overcurrent Set oint:
The licensee determined that although the ASW pump motor overcurrent trip setpoint had not been appropriately changed with the change of the pump impeller, the pump would not trip on overcurrent during a degraded voltage condition due to the designs of the offsite electrical power supply and onsite emergency power supplies (in regards to extended undervoltage protection).
Diesel Generator Fuel Oil Consum tion:
The licensee determined that even when considering the additional load requirements on the diesel generators, the TS fuel storage limit remained greater than a new seven day fuel consumption calculation.
The licensee stated the
'ncreased consumption could be accommodated within the margin in the calculation.
During the April 6, 1989 meeting the differences in the calculations were discussed.
The licensee's new calculation recalculated the amount of fuel oil available based on the actual suction pipe level, considered an ASW motor horsepower of 440 H.P. to be more representative than 450 HP but maintained high loads on all running equipment.
The licensee's new results do show the technical specification minimum gallonage was sufficient to accommodate seven days of diesel operation as calculated using the design basis.
The licensee also demonstrated that with actual, maximum expected loads and normally available fuel inventory, that 3.0 to 3.8 days of fuel supply was calculated.
The licensee also stated that a records search showed the lowest fuel inventory achieved was 63,000 gallons of fuel since the impeller changes were made.
The licensee also stated that many sources of fuel were available, which included on site diesel fuel storage for auxiliary boilers and storage facilities in Avila Beach, which assure delivery within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />.
Root Causes Root Cause for the Incorrect ASW Overcurrent Set oint:
The licensee determined that Engineering had not properly identified that a
review of the motor performance test was required to establish the overcurrent relay setting prior to pump operability.
The LER also identified the following contributory causes:
1.
A notation to check the pump overcurrent relay settings was made on the ASW pump motor rerating calculation.
However, Engineering failed to designate the calculation as preliminary in accordance with Engineering Manual Procedures.
2.
The pre-operational test requested by Engineering did not specify testing to simulate worst case conditions.
3.
The Engineering instructions on Replacement Parts Evaluation (RPE), that stated the need for motor test data, were incomplete and unclear.
4.
, The technical review checklist included in the Engineering design change procedure did not include a specific entry for motor overcurrent relay setpoint evaluation.
Diesel Fuel Oil Consum tion:
The LER states that the engineer failed to review the fuel oil consumption calculation for impact from the pump design change.
Contributory causes were identified to be:
1.
The safety evaluation for the design change did not recognize the need to.revise the fuel oil consumption calculation.
2.
The technical review checklist included in the Engineering design change procedure did not include a specific entry for diesel fuel oil consumption.
Corrective Actions:
With respect to the ASW overcurrent setpoint, the licensee performed a justification for continued operation (JCO)
CFR Part 50.59 safety evaluation on February 2, 1989, to justify operation until the setpoint was raised from 67.5 amps to 75 amps on February 4, 1989.
With respect to diesel fuel oil storage, the licensee revised its fuel consumption calculation, performed a
JCO on February 2, 1989, and revised surveillance procedures to increase the acceptance criteria for quantity of fuel oil inventory.
As corrective actions to prevent recurrence, the LER specified the following:
l.
All test results required by Electrical Engineering on Mechanical replacement part evaluations (RPEs) will be reviewed to verify that any required test data has been furnished and is acceptable.
In addition, a twenty percent sample of all Electrical RPEs issued following commercial operation will be reviewed.
If any problems are noted, all Electrical RPEs will be reviewed.
2.
All Electrical design change calculations will be reviewed to ensure that, if needed, they are designated as "Preliminary" until appropriate input is received and they are reviewed and finalize.
The need and importance of the following areas will be reemphasized to design engineers in training sessions.
a)
Purpose and use of preliminary calculation procedures.
b)
Clear instructions for required test data.
c)
Proper use of RPEs.
d)
The need to consider and document all interactions between associated systems and components when performing a 10 CFR 50.59 evaluation, even though interactions are initially judged to be insignificant.
4.
Training for Engineering Group Supervisors and Group Leaders will be conducted to address the importance of identifying all affected calculations and of looking for other factors that are outside the obvious impacts of the design change.
5.
The Engineering design change procedure was revised on March 2, 1989, to ensure that requirements for component and system testing are identified by the engineer writing the design change, including requirements to provide for design verification of the system following modification.
6.
Revise the design change procedure to add the subject of diesel fuel oil inventory and relay settings to the technical review checklist.
7.
Based on the above NRC concerns, combined with the licensee decision to add a sixth dedicated diesel generator, PG8E is evaluating increasing fuel storage capacity at DCPP for the DG system and raising the Technical Specifications limits.
Failure to Reinstall Backwater Check Valves in Fuel Oil Transfer Pum Vault Drain S stem LER 50-275/89-02 Note:
This LER was not in response to the NRC team inspection 50-275/89-01 but is in response to the resident inspector report 50-275/89-05 and is included because of the importance of timely analysis.
On March 27, 1989, the licensee submitted LER 50-275/89-02 regarding the failure in 1978 to reinstall backwater check valves in the diesel fuel oil (DFO) transfer vault drains following the relocation of the vaults.
Previously in September, 1988, an NRC maintenance team inspection (Inspection Report 50-275/88-15)
issued a notice of violation for the lack of a testing program for the DFO transfer pump vault backwater check valves.
On December 22,, 1988, while performing an inspection of the DFO transfer pump vaults to determine a testing method, maintenance personnel discovered that the backwater check valves were not installed in the drains.
Although maintenance
requested an engineering evaluation, it was not until February 24, 1989, when the resident inspector questioned the issue, that engineering informed the plant that the absence of backwater check valves represents a condition that in combination with other events could prevent fulfillment of the safety function of the diesel generators.
~Anal sis:
The licensee's LER analysis restated the analysis presented in their October 5, 1988, response to the notice of violation.
In summary, it stated that given 1)
a plugged drain line downstream of the drain header for the DFO transfer pump vaults and the ¹2 heater drip pump vault, 2) a catastrophic failure in the circulating water or condensate system causing the flooding to 10'bove the floor in ¹2 heater drip pump vault, and 3) the loss of offsite power, the lack of backwater check valves could than cause the DFO transfer pump vaults to flood and make the DFO transfer pumps inoperable, resulting in the loss of fuel to the diesel generators.
The licensee's immediate corrective action determined that the drain line was free of blockage and that the DFO transfer pump vault water level detectors (which annunciate in the control room) were operable and accurately calibrated.
The inspector found the analysis to be acceptable.
Root Cause:
The inspector found acceptable the licensee's determination that the root cause was inadequate instructions provided to contractors during the 1978 reinstallation of the DFO transfer pump vault.
The licensee identified as a contributory cause the lack of quality control verification of the reinstallation.
Corrective Actions:
On February 24, 1989, the licensee prepared a
justification for continued operation and a 10 CFR 50.59 review which identified interim compensatory actions such as blocking the
¹2 heater drip pump vault drains and periodic verification of diesel fuel oil transfer vault water level.
The LER identified the following corrective actions to prevent recurrence.
Backwater check valves were installed in the diesel fuel oil (DFO) transfer pump vault drain.
2.
To ensure the design bases are maintained, requirements for the DFO transfer pump vault backwater check valves will be included in the Design Criteria Memorandum (DCM) for the diesel generator system.
3.
The DFO transfer pump vault backwater valves were added to the preventive maintenance program.
Procedures were reviewed and the determination was made that adequate instructions are currently provided to PG8E personnel to ensure that sufficient guidance is provided to contractors performing work activities in support of PG8.
An expeditious review of the FSAR and NRC Safety Evaluation Reports (SERs) is being performed by system and design engineers to ensure that the design bases are appropriately implemented in plant procedures.
This review will be completed by June 30, 1989.
A more thorough review of the FSAR design bases is being performed during the development of the DCMs.
6.
A Configuration Management Program was initiated in late 1988 to incorporate detailed system design basis information into existing DCMs and prepare additional DCMs as necessary.
Twenty-one DCMs are scheduled to be completed in 1989.
After completion of a DCM, a detailed review of plant procedures will be reviewed to assure the proper incorporation of system design bases.
Timeliness of Problem Identification:
The LER recognized that prompt Engineer ing evaluation and corrective actions had not been accomplished.
The cause was determined to be the maintenance engineer's fai lure to correctly communicate a problem through 'the chain of command and the failure of Engineering to recognize that a
system design basis question needed to be pursued expeditiously.
The adequacy of the licensee's corrective action for timeliness will be evaluated in future inspections.
On March 17, 1989, an exit meeting was conducted with the licensee's representatives identified in paragraph 1.
The inspectors summarized the scope and findings of the inspection as described in this report.