ML20205L107
| ML20205L107 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 02/14/1986 |
| From: | Ray H SOUTHERN CALIFORNIA EDISON CO. |
| To: | Martin J NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| References | |
| NUDOCS 8604030117 | |
| Download: ML20205L107 (19) | |
Text
"'
1
,&J&i:a 3eb &L~
- G'
.e-e
.+.v.
a t
,f we.1" RECElVED i
y y
l Southem Callknie Edison Company Y
E6 gen ig Mt G 59 l
^ o = ox '= =
SAN CLEMENTE. CALWORNIA 92672 HAROLD B. RAY vice pecsiormt a site MANAGE.
cTELEPHoNE February 14, 1986
. %W "ri+=.-.A7o i
.A=o=or.c I
i i
Mr. John B. Martin, Regional Administrator i
U.
S. Nuclear Regulatory Commission 1
Region,V Office
}.
1450 Mhria Lane, Suite 210 l
Walnut Creek, CA 94596-5368
Dear Mr. Martin:
i.
t During our discussion on January 31, I indicated
[
that I would look at the facts concerning San Onofre's experience with the 5 check valves that~were found failed after the November 21 event, and let you know the results.
This turned out to be a rather big job.
t l'
The results are contained in a memorandum to f'
L i
Larry Papay dated February 14, 1986.
I found~I had I
to understand the records management picture over the past 15 years as well, and the results of that I
investigation - in terms of IIT Finding #20 conclu-i sions - are contained in a similar memorandum dated February 11.
Copies of these memoranda are attached E
for your information, and I have provided copies to Randy Huey.
3 l
6kbNY f
Attachments 1
i f
i i
96 Wg r
mmm wayg.
cp' s
ii Yl
-}
U 1,
- N :'
February 11, 1986 DR. L. T. PAPAY
SUBJECT:
Status of Review of Maintenance-Praccices and Records San Onofre Nuclear Generating Station, Unit 1 PURPOSE The following status is provided concerning our review of one of the central issues that appears to have emerged from the IIT report.
The principal basis for the criticism of maintenance practices at SONGS, and the implied criticism of the overall management process, is the IIT finding that:
~
- 20. Station maintenance records are incomplete, difficult to locate and, when available, lack sufficient detail to determine what was done.
The assumption is that, if our records are this bad, then maintenance and management practices must.be as bad, or worse.
Unfortunately, in order to understand this. issue, it is'necessary to examine some details.
As indicated in my memorandum to you dated February 7, 1986, we have a comprehensive report in preparation on what is now identified as NRC Item 5, and it will be completed by the end of this week.
The purpose of this memorandum is to provide a summary of its findings.
SUMMARY
I San Onofre maintenance records not only meet regulatory requirements, but the process whereby they are generated has l
recently been noted by INPO as a standard of excellence for the industry.
The IIT report finding is misleading, in that it would appear to be based on existing requirements.
However, from a conversation with the IIT leader, it seems to have been based l
instead on a misapplication of an obscure provision in the ANSI
' standard, which is also not accepted by the industry generally.
l l
. =
I e'.
,e,
=
8 DR. L. T. PAPAY February 11, 1986
[
DISCUSSION Record Retention
~
The FSAR states that, " Records of quality assurance activi-ties associated with the equipment listed in Table D.1 are retained for a minimum of 6 years."
(As discussed in a separate memorandum, 3 of the Feedwater System check valves of interest are clearly not in Table D.l.)
This commitment was made in 1972, prior to the issuance of the applicable ANSI standard in 1974.
Until recently, regulatory requirements imposed on most i
plants, including San Onofre, were based on ANSI N45.2.9-1974.
This requires that records which involve " Principal Maintenance Activities, Including Inspection, Repair, Substitution or Replacement of Principal Items of Equipment Pertaining to Nuclear Safety" must be retained for only 5 years.
(Emphasis added.)
In an August 1985 revision, the Regulatory Guide that implements QA Program requirements for the design and construction phase I
included a lifetima retention Deriod for " Records and logs of maintenance activities, inspections, repair and replacement of principal items...."
(Naturally, this August 1985 revision does not apply _to San onofre, but its objectives may 'nderlie some of u
the criticism we have received.)
An appropriate strategy with the IIT might have been to say l
'that we do not keep maintenance records longer than J years (6 per the FSAR) and that anything generated prior to 1979 would not I
be readily available.
We didn't do this, of course, and we made i
an attempt to produce what we could The fact that this was a difficult job, as it would be at all older plants, was not at all j
iappreciated by the IIT.
,g i
l Record Retrievability i
Unit 1 maintenance record generation, filing and retrieval are now fully integrated with the system 2stablished originally for the construction and startup of Units 2 and 3.
After a time, that system was applied to Unit 1 backfit and modification, and j
.only recently it was applied to Unit 1 operation and maintenance.
The system is fully computerized, and we think it is the best in
{
the nation today.
However, retrieval of Unit 1 maintenance records generated N_ior_to the 1984 Return _to_JSerVice_2rogram_has_.to_take the-following into consideration:_
J o
As indicated.above, we aren't even required to keep records i
longer than 6 years.
We do not intend to. expend the effort j
required to sort prior records, of whatever age, and place 4
.------,,...,..-,,,,,.--,--,,-,-.-,-..,,--,,--,-,-,,.-.,-,,---..--,,,--,---,,,-,,__s-.
i.
- *.." A 4.
g *c DR.-L. T. PAPAY February 11, 1986 them into the current retrieval system.
If sufficient cause exists, as in this case, we can go through these old records, which were filed in a number of different ways, and pull out what comes to hand.
But we are only now in a position to sa'y that we think we have most all of the older
- pecords that exist onsite for the 5 check valves.
(We will-(never be able to say that we definitely have all the-records.)
o Unit 1 did not have a plant tag system that identified each component, until recently.
For tnose cumyonents that did have an identifier originally, there is now a new tag i
number.
Today, all work can be related to a unique component i
tag number, but, in the past, documentation of work could be filed in many different ways.
(As a simple example, check 4
valve records in the past have been filed under the manufac-turer, the system and the function noun-names, as well as the original valve numbers which were the same for similar valves.)
o As indicated above, the transition of Unit 1 records tx) the current system first involved work done by the Project and later work done by the Station.
Thus, in addition to multiple identifiers for retrieval'of older' records, one murt look in two separate sets of records:
those generated by the Project and those generated by the Station, both of whom performed " maintenance" work at various times.
Both of these separate sets have different locations for records involving maintenance, inspection, testing, etc.
Contents of Records There is a tendency to confuse " maintenance" records with other records, such as " inspection and test" records.
In our discussions with them, the IIT appeared to expect the results of investigations done by maintenance personnel to produce records equivalent to those required for inspections and tests by QC and engineering personnel.
We fail to find any basis for such an expectation, either in the regulatory requirements or in industry practice.
ANSI N45.2.10, " Quality Assurance Terms and Definitions,"
includes the following:
o Inspection - A phase of quality control which by means of examination, observation or measurement determines the conformance of... components...to predetermined requirements.
DR. L. T. PAPAY February 11, 1986 ANS-3.2/ ANSI N18.7-1976, " Administrative Controls and Quality. Assurance for the Operational Phase of Nuclear Power Plants," contains the same definition of inspection.
Further, it clearly separates " maintenance" from " inspection" throughout.
Thus, requirements for maintenance records are not the same as the extensive requirements for inspection records.
'The IIT report claims that station maintenance records
... lack sufficient detail to determine what was done."
After issuance of the report, Mr. Tim Martin stated orally that this deficiency is not limited to our older maintenance records, but is also true of. current records.
However, it seems clear that what he is really looking for is not what maintenance was done but information that would accurately disclose why it was done.
He said that San Onofre records do not satisfy the requirements of ANSI N 18.7 that we have the capability to detect " trends" in equipment performance, and neither do records for other nuclear power plants.
The requirement to which Mr. Tim Martin refers appears in Section 4 of ANSI N 18.7, where it provides that:
" Programs for reviews and for audits of activities affecting plant safety during the operational phase shall be established by the owner organization to:
(4) Detect trends which may not be apparent to the day-to-day observer" The context in which this requirement appears is that of OSRC, ISEG, etc.; and the extensive discussion that appears elsewhere concerning the creation and use of records makes no reference whatsoever to maintenance records including information to support this function.
Over the past two weeks, we have surveyed 13 separate utilities.
The results of that survey will be documented in the report referred to earlier, but suffice to say here that:
(1)
San Onofre meets or exceeds the level of maintenance record completeness that represents accepted industry practice, and (2) both San Onofre and other sites have vastly increased the amount l
of information included in maintenance records.
To illustrate the latter point, the number of data blocks on a San Onofre maintenance order has increased from 18 in 1977, to 50 in 1978, to 118 in 1980 and to almost 140 today.
Similar
~
increases have occurred at other sites.
Even so, it is probably true that they do not include the sort of information to which Mr. Tim Martin alluded, as discussed above, and there is no i
regulatory basis for them to do so.
J
t 5..
..,e DR. L. T. PAPAY February 11, 1986 Completeness of Maintenance Records The IIT report claims that, " Station maintenance records are incomplete...."
The ordinary interpretation of this finding
-would be'that required entries on the records are missing, or that records known to exist, or which should have existed, are missing.
It would be foolish to say that, in the literally hundreds of thousands (probably millions) of entries in mainte-i nance records, each and every entry is complete and perfect.-
However, except.for the issue discussed above, conceraing information Mr. Tim Martin thinks should be provided to support i
trend analyses, we are not aware of a lack of completeness in these records that would justify this IIT finding; to the contrary.
current requirements provide for review of maintenance records by five individuals:
(1) the journeyman, (2) the foreman 4
or supervisor, (3) the general foreman or supervisor, (4) main-tenance scheduling and (5) maintenance planning.
This does not include review by QA/QC.
Each individual is not looking at the entire record for the same thing, but it is not credible or true j
that_our records generally lack completeness.
In this regard, the 1985 INPO evaluation of San Onofre included Good Practice MA.3-1 which includes the following:
i "The San Onofre maintenance program is significantly enhanced by the use of a computer-generated, integrated i
maintenance planning program.
For any specific plant component, this program will identify items such as the following:
a.
Outstanding preventive and corrective maintenance orders j
that are ready to work.
b.
Operating modes required to perform the maintenance.
c.
Quality assurance and control requirements applicable to the maintenance to be performed.
l d.
Outstanding maintenance orders that are currently being planned."
As you know, a Good Practice is publicized by INPO to the industry as a standard of excellence.
Although the foregoing quotation'does not refer directly to the " completeness" of the resulting records, they were extensively reviewed by INPO in the i
process of generating this Good Practice, and they were found to i
be complete.
Otherwise, INPO would not have referrea the industry to our program as a standard of excellence.
.g
, f.g :. -
DR. L. T. PAPAY February 11,'1986 Specific Records Concerning Feedwater Check Valves The IIT report contains a number of references to " apparent discrepancies," inability to identify "other completed work orders" which were assumed to exist, etc.
In summary, no prob-lems with these records exist, in fact, and this is discussed in a separate memorandum concerning_the check valves themselves.
CONCLUSION Because the integrity of our entire maintenance program, and of our management process generally, has been challenged by the IIT criticism of our maintenance records, we have devoted much attention to determining if the criticism is warranted.
In order to avoid parochialism and defensiveness, we have involved outside resources in examining the issue.
The conclusion is that the finding is not_ appropriate, at least as it would be understood by anyone unfamiliar with the unique perspective of the IIT and the details of this situation, and that we should find some way of correcting the significantly adverse impression it has created.
Fina'lly, there are so many recordr, which have been created under different programs over more than a decade, that we clearly cannot claim that they are all perfect, even though we are not aware of any significant deficiencies.
4 If you have any questions or comments, or if you would like additional information concerning the above, please let me know.
i HAROLD B. R l
HBR:jkb I
cc:
Kenneth P. Baskin Harold B. Ray l
Glenn J. Bjorklund l
Site Management CDM i
1
+-
,,--r,---
.,..,., ",,m
.me-,#.,,mw7
-4.m-,m,,.,,,
e-,.mmwm.,,-_v,w,
-,--.,--mm,%.-m,-,,m,.,.y..,y,c,..-,v,.
7 ww
,m.--,%,~,
w.,m
F, 3;
T,. ',}.
February 14, 1986
,DR.:L..T.
PAPAY
SUBJECT:
Experience With 5 Feedwater System Check Valves San.Onofre Nuclear Generating Station, Unit 1 PURPOSE The IIT' report reaches no conclusion'concerning the cause-of
- failure of the 5 Feedwater System check valves.
In a memorandum to you. dated January 28, 1986, I summarized the results of our determination of the cause.
However, in'a meeting with Region V on-January 31, 1986, Mr. Martin indicated that he had inferred from the.IIT report data that we had experienced extensive and-chronic problems with the valves.
In addition, Mr. Stello's letter of February 4 includes, as Item 1, an investigation of the performance of check valves at San Onofre, including the performance of these particular check valves.
As a separate matter, the IIT report includes a finding;that-Station maintenance records are incomplete, difttcult to locate and,.when available,' lack sufficient detail.
This is included in Mr. Stello's letter as Item 5, and, with respect to maintanance records generally, I have provided a summary of our findings in a memorandum to you dated February 11.
However, that memorandum does not include information from the records that is specific to the 5 valves.
The purpose of this memorandum is to summarize the results of out. investigation into our experience with the 5 Feedwater System check valves of interest, in partial response to NRC Items 1 and 5.
SUMMARY
Because of many differences between the applications, the three 10-inch feedwater regulator discharge check valves must be
-discussed separately from the two 12-inch MFW pump discharge check valves.
The three 10-inch valves experienced several significant probleins up through 1976.
They were modified by Steam Division maintenance forces during the 1977 refueling, and they subse-quently appeared to have performed well for almost 10 years until
x DR. L.
T.
PAPAY February 14, 1986 the November 21, 1985, event..
They were inspected during the 1978 and 1980 refuelings and passed numerous inservice tests commencing in 1978.
Until relatively recently, these valves were not considered by the Station to be safety-related.
The two 12-inch valves appear to have experienced only one problem prior to the November 21 event.- In 1979, a failure occurred that may have been similar to that which occurred in 1985.
However, the valves were subsequently inspected and tested with no' indication of problems.
The check valves were modified in 1981 to include holes in the discs, so the inservice testing subsequent to that date only confirms the absence of gross disc failure.
Thus, it was reasonable for the Station to believe that the problems experienced previously were either isolated occur-rences, or they had been corrected by modifications, and that they would not recur.
DISCUSSION OF SAFETY-RELATEDNESS The IIT report explicitly assumes that the five check valves both are, and have been, considered to be safety-related.
This is not entirely true for the three feedwater regulator discharge check valves, and this fact is significant to an analysis of our experience with the valves.
FSAR Ouality Assurance Equipment List Table D.1 in the FSAR identifies the items in the " Quality Assurance Equipment List."
It clearly does not include the three check valves downstream from the feedwater regulator valves.
Instead, the feedwater pumps, and the piping and valves associ-ated with their safety-related functions (including the two discharge check valves), are listed aa part of the Safety Injection System.
The table itself has been revised 7 times since original issue, most recently April 25, 1984, with no additions dealing with the Main Feedwater System, or its components.
Table D.1 does list the Auxiliary Feedwater Pumps as part of the " Emergency Power System", and it also includes " Piping and valves required to deliver water to the Auxiliary Feedwater Pumps from.the Condensate Storage Tank and the Service Water Reser-voir."
It does not include any reference to the discharge side of the AFW pumps, or to its connections with the Main Feedwater System.
(Also, the Feedwater System is not included as part of the Containment Vessel and Penetrations, since it is considered to be a closed system inside containment.)
u
- .5 t
DR. L. T.
PAPAY February 14, 1986 Station Order S-A-112, Station Quality Assurance Program First issued December 28, 1972, this order was revised 6 times, most recently June 24, 1982.
Initially, the "Q" List it contains was identical to the FSAR.
However, Revision 2, issued September 20, 1977, includes the following addition:
Condensate and Feedwater System o
All systems not associated with Safety Injection or Auxiliary Feedwater System:
"Q" Class - No o
Components, piping, and valves associated with SIS:
"Q" Class
- Yes The three check valves downstream from the feedwater regula-tor valves were not considered by the Station to be " associated with the Auxiliary Feedwater System" and therefore - even after 1977 - the valves continued to be considered non-safety-related.
This has been confirmed by discussion with personnel who were cognizant plant engineers during this period. (See, however, the discussion of Code applicability and inservice testing below.)
Tne check valves were not considered safety-related, even though one of the two AFW flowpaths was through the first point feed heater and then through these valves, because that was considered to be the non-safety-related AFW flowpath.
The function of the va.-
's in preventing AFW backflow during a loss of inplant AC power c/ent, when the MFW pumps were stopped and the MOV block valves remained open, was not considered to make them safety-related.
In fact, as noted above, the entire high pressure feed system was not initially classified as safety-related, except for that portion involved in the safety injection j
function.
l In 1979, NRC inspectors reviewed this Station Order against the FSAR and cited several omissions, not including anything to do with the Main Feedwater System.
Drawing M-37560, San Onofre Unit 1 Q-List By a memorandum from Mr. Pilmer to Mr. Haynes dated May 10, 1984, the responsibility for the Q-List was transferred from the Station and incorporated in a controlled drawing maintained in i
accordance with an E&C QA Procedure.
Initially, the list was the i
same as in the Station Order, but Rev. 1 of drawing M-37560, issued in August 1985, was a greatly expanded and more detailed listing than had ever existed previously.
It lists all Feedwater System piping outside containment, to and including the MOVs, as safety-related.
i l
l-
DR.
L. T.
PAPAY February 14, 1986 As indicated below, the three check valves downstream from the feedwater regulating valves had~been considered Code, had been included in the inservice testing. program, and were in piping that was shown on the P&ID as safety-related somewhat earlier (1980), but this appears to be the first time they were included - either directly or indirectly - in a Q-List.
Piping and Instrument Drawings By a Configuration Change Notice (CCN 6) issued September 5, 1980, all five valves were included in piping designated as saf.ety-related'on the P& ids.
Prior to this, the P& ids generally did not provide safety-related/non-safety-related boundaries.
The earlier P& ids did, however, show the components that were subject to the " Code," referring to the Boiler & Pressure Vessel
~
Code; the same as for a fossil-fueled steam plant.
Therefore, we find that the three check valves were subject to some controls because they were " Code" (Section I or ASA B 31.1).
But, they were not generally viewed as falling under the provisions of 10 CFR 50, Appendix B, until quite recently.
Inservice Testing Program Requirements for inservice testing in accordance with Section XI of the ASME B&PVC were developed in mid-1977 and submitted to the NRC in September to become effective in 1978.
Secticn XI presumes that the piping systems have been classified in accord-ance with the design requirements of Section III.
Unit 1, of course, was not designed to those requirements, and it was first necessary to determine where the Section III Code Class boundaries should be placed for the existing systems.
In new designs, the Code Class boundaries and the safety-related/non-safety-related boundaries generally coincide.
For Unit 1, this is not the case in some instances.
Also, the determination of the Code Class boundaries, and the resulting implementation of the inservice test program, occurred completely independently from the designation of safety-related components and systems, at least insofar as the Station was concerned.
Since the Section III/XI scheme didn't fit the Unit 1 design anyway, the Station did not consider it necessary to reconcile what would have been inconsistencies of classification for a new design plant.
Therefore, even after implementation of inservice testing, and the designation of the piping in the area of the three feedwater regulator discharge check valves as Code Class 2, the piping was not shown on the drawings as safety-related until 1980 l
l l
J
,e_-
o n
DR.
L. T.
PAPAY February 14, 1986 and it was not included in the Q-List as safety-related until 1985.
Even though the three valves were subject to inservice testing in 1978, they were not necessarily consiaered to be safety-related~until so designated on the drawings and/or included in the Q-List.
Understanding this background is crucial to evaluating the maintenance experience discussed below.
DISCUSSION OF MAINTENANCE EXPERIENCE IIT Report Findings Concerning Experience With the 5 Check Valves The'IIT report is based on the information that could be retrieved in the time available.
(See my memorandum dated February 11 concerning records in general.)
The IIT report findings summarize that information as follows.
(The statements in parenthesis represent additional information that has been obtained from the data files since input to the IIT was no longer possible.)
"The MFW pump discharge check valves were last disassembled, inspected and reassembled on May 5, 1980.
All internals and seating surfaces were reported to be in good condition.
No additional recorded maintenance since that time has been found."
(As a result of the failure to open of the Safety Injection System hydraulic valves, the MFW pump discharge check valves were modified by the Project during the 1981 outage resulting from that event.
The discs were carefully inspected at this time and there were no problems.
The replacement internals included a hole that was drilled in the disc to permit limited backflow through the check valve.)
" Maintenance on the feedwater regulator check valves dates back to the SONGS-1 refueling outage in 1975."
(We now have obtained records that show maintenance performed as far back as 1973.)
"The maintenance history can be summarized as follows:
"1975 Refueling:
Inspected three regulator check valves and installed new internals in all three check valves.
"1977 Refueling:
Inspected and installed new internals in all three check valves.
(The internals were actually modified by Steam Division maintenance personnel at this time.)
I a
g T
+
DR.
L.
T.
PAPAY February 1.4, 1986 "1978 Refueling:
Inspected all three check valves,sfound no problems, cleaned parts, and reassembled valves.
"1980 Refueling:
Inspected all three check valves, replaced cotter pins, washers and nuts on flappers, and reassembled.
(Additional detail is provided in the discussion of the work done below.)
"SCE personnel could not identify any other completed maintenance work orders for any of the check valves-in the feedwater system."
(We have subsequently retrieved a number of additional work orders.)
s Our experience with the check valves is summarized below.
Feedwater Regulator Discharge 10-Inch Check valves The earliest piece of data recovered so far is for the B gy generator check valve.
In December 1973 a repair was performed which is described as follows:
"B steam generator feedwater check valve has an unusual internal noise.
Repaired."
This is documented on Form PSSD-104, " Request for Equipment Repair."
The repair occurred during a non-refueling outage which was taken for other reasons.
In July 1974, while raising power following a shutdown, a restriction in feedwater flow to the C ateam generator was identified.
While operating at 150 MWe on the bypast, the regulator discharge check valve was opened.
The recotd shows that, "The flapper of the check valve was found detached from the 2(
arm and lying in the check valve body thus causir.g the flow restriction."
The valve was repaired and the unit taken to full load.
The record states that the Station discussed the event A
with the NRC (AEC) Regional office and,
"...it was agreed this was a non-reportable item.
The valves were next opened for routine inspection during the 1975 refueling outage, as noted by the IIT.
A Steam Division BSC Work Crew Summary of that outage indicates that the internals of all three check valves were replaced and that, " Extensive wear X
was noted on the hinge pins and hinge arms."
There is no indication that the check valves had failed, or that they were about to fail.
s, Operating logs show that on March 19, 1976, unit load was
's reduced to 110 MWe
"...to replace the internals in the C steam generator feedwater line check valves (sic.)."
Load apparently was reduced for about 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> to perform this operation.
There l
1
DR.
L.
T.
PAPAY February 14, 1986 is no information concerning why this was done, or what symptoms there were of thic second failure of the C check valve.
On April 19, 1976, while returning to power from a unit trip, power could not be increased above a certain level due to a restriction in feedwater flow to the B steam generator, identical 3f to that which had occurred 2 years earlier to the C steam generator.
Investigation revealed a noise in the vicinity of the check valve.
The valve was cleared, opened and repaired, and the unit was returned to service within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />.
The cause of the check valve failure was determined to be,
"...the disc holding pin had broken, allowing the disc to fall into the bottom of the valve."
Repair consisted of installing a new arm, disc and pin.
x [f
.Following the return to service on April 20, the hinge pin plug
\\r was found to be leaking steam and it was weld-repaired while on
// ['
the line.
(This was a rather common occurrence, and not all such instances are included in this discussion.)
No maintenance records concerning the work done on March 19 and.'.pril 19 have been found, consistent with the fact that these valves were'not considered by the Station to be safety-related in 1976.-
Work was done on all three valvea during the 1977 refueling outage (which began in October 1976).
Again, as in the 1975 outage, the only record of the work is in a Division Maintenance Boiler Section Work Summary Report assembled at the completion of the outage.
The report states that, "New internals were installed in these valves with anti-rotation clips to prevent the
//
disc from turning."
(Emphasis added.)
Although it is not indicated in this record, we believe this is when the nuts were also drilled and pinned.
Important points to be noted here:
I Work on the valves during refueling outages was being done at o
I this time by Steam Division maintenance personnel, who j
presumably were bringing to bear the experience they had with this equipment generally.
Therefore, the application of the anti-rotation clips may o
have been in response to the earlier problems at Unit 1, or it may have been in accordance with developing system-wide experience.
l o
As will be shown below, the performance of the three regulator check valves was without significant problem for almost 10 years following this modification.
This is extremely important, because it would have appeared that the l
l l
~
DR. L. T.
PAPAY February 14, 1986 i
modification performed by Steam Division maintenance personnel in 1977 had cured the earlier problems.
L As indicated above, the Section XI inservice testing program l
was implemented in 1978.
From that time until the November 21, 1985, event, many tests were performed on the three regulator check valves with no tests recorded in which any of the valves were unable to. meet the acceptance criteria in the procedure with the steam generator at pressure.
1 1
It is extremely important to: recognize that the inservice j
test required by Section XI is a demonstration that the valve i
closes.
It is not a seat leakage test.
The procedure which initially implemented the inservice testing program provided for testing the check valves when seated by only the static head of water,from a cold, depressurized steam generator.
The test acceptance criterion was " minimal.or no flow".
This is obviously a subjective test, but one that is entirely adequate to demonstrate the disc has moved from its normally open position to i
i a closed position.
l However, since the leakage is not required to be measured, it I
is not possible to determine if leakage at this low backpressure changed over the 6 years (including.two extended outages) that testing was conducted between'1978 and the return to service testing in 1984, or if the measure of what people considered to be acceptable changed.
In either case, commencing in 1984, and continuing into 1985, the backleakage was usually not judged to be acceptably low until steam generator pressure was increased 4
during plant heatup.
(Again, note that the demonstration required by Section XI is one of valve closure; not leaktightness of the seat.)
Therefore, in May 1985, the procedure.was changed to require that the test be done with the steam generator at pressure, since this then avoided the uncertainty that had l
existed concerning what to do with the.results of static head testing.
j' Again, during the 1978 refueling (which began September 18, 1978) the three regulator check valves were inspected by the Division Maintenance Boiler Section.
Their report states:
" Main Feed Water Check Valves "a.
The valves were disassembled and inspected.
"b.
Installed new orifices, springs and plugs."
1 i
Since the 5 check va)ves do not contain orifices or springs, i
this reference is not clear.
However, in a daily, handwritten log we find the notation:
" Main FW Ck V1vs - All apart, found no problems.
Will clean parts and reassemble."
t u
w r
DR.
L. T.
PAPAY February 14, 1986
. Finally, the Division Maintenance Boiler Section Summary Report for the 1980 refueling outage states for these valves that, "All. valves were disassembled.
And (sic.) the parts were cleaned and inspected.
No problems were found.
All valves were reassembled."
Handwritten notes made at the same time read as follows:
" Removed covers, inspected seats.
Replaced flapper back-up washers.
Drill out pin on 456 (A).
Tap and die all flappers and nuts.
Redrilled 456 (A) flapper nut.
Replaced seal rings.
Cleaned all parts.
QC inspected repairs."
(Note: 456 (A) is the regulator valve.
Records for the check valve were frequently recorded and filed with the regulator valve.
The regulator valve for the A generator is FCV-456.
See my February 11 memorandum concerning record retrieval generally.)
The most recent inservice tests recorded were performed on the regulator discharge check valves in February 1985 with the steam generators at pressure.
All three passed successfully.
MFW Pump 12-Inch check valves As discussed earlier, these check valves have been designated both safety-related and Code from the beginning.
The refueling outage work may have included inspection of their condition, along with'the three valves discussed above, under the general heading " Main Feed Water Check Valves."
However, all repairs described in the records clearly relate to the regulator discharge check valves, and not to these two valves.
The only record of any problem with these valves appears in April 1979.
The East MFW pump was stopped with the West MFW pump
+
- running, The East pump began t0 rotate backward, and the discharge block valve was immediately shut accordingly.
The records relate only to the inspection and subsequent test of the MFW pump to determine if it had been damaged by the reverse rotation.
However, the recollection of personnel here at the time was that the check valve was opened, and it was found that there was " excessive play" in the flapper assembly.
New internals were withdrawn from stock, and the check valve and pump was returned to service.
Some personnel vaguely recall other symptoms of possible check valve backleakage prior to 1981, but specific tests of valve leakage showed ther were tight.
(Integrity of these check valves has always been considered very important, for pump protection reasons.)
The cognizant station engineer recalls that it was decided to inspect both check valves during the 1980 refueling outage, as a result of this event.
The valves had operated for about a. year following the April 1979 event when they were both inspected in May 1980.
The maintenance records for each valve indicate, "All internals and seating surfaces good."
The East pump valve had the seats blue-checked.
a.'
DR.
L. T.
PAPAY February 14, 1986 Shortly after the unit returned to service from the 1980 refueling and steam generator repair, it tripped with SIAS actuation, but the hydraulic SI discharge valves failed to open.
Resolution of the underlying problem was undertaken by the Project, and resulted in numerous design changes.
These included replacement of the check valve discs with new discs which con-tained drilled holes.
The valves had operated only briefly following the inspection descritted above, but they were carefully inspected when removed and there were no deficiencies.
These two valves were subjected to inservice testing starting in 1978, in parallel with the three regulator discharge check valves discussed above.
Initially the test consisted of pres-surizing the valve using the AFW pump and observing for any backleakage.
A total of 6 such tests were conducted in 1978 and 1979 (3 on each valve), and all passed.
However, a revision was then made to the inservice testing program which provided for r
these valves that:
"No test required.
The normal position of this valve is also its safety-related position."
e
~
This was revised again in 1984 to reinstate the requirement that the valves be tested.
However, by this time the design change had been made to provide a 1/4-inch hole through the disc J
and to also cut the seat.
Thus, the valve currently is far from leak-tight under design conditions.
In order to pressurize the disc with this design backleakage, it was decided to use the y
opposite MFW pump, instead of the AFW pump.
The only thing this
/
E test can detect is gross failure of the disc.
Such failure would
-~ {
j t
be manifest by reverse rotation of the pump and blowing of the
/
seals, as occurred on November 21, 1985.
The test has been conducted once:
during the November 1984 return to service testing.
Because of the hazard inherent in conducting the test, Operations is naturally reluctant to conduct the test in this manner and is looking for some alternative test method.
Noise Heard In B Feedwater Line, June 1985 In June 1985, a " rapping noise" was heard in the breezeway between the turbine building and the containment.
An immediate s
1 investigation =was performed which located the noise to the B feedwater line, seemingly in the vicinity of the manual block valve, downstream from the regulator check valve.
The following action was taken.
The block valve was radiographed, exercised to a partially o
closed position and the vendor was contacted for information concerning the possible causes.
b b
r 2
=
.e DR.
L. T.
PAPAY February 14, 1986 o
vibration and stethoscope measurements were taken at the block valves, the check valve and elsewhere along the line.
These measurements indicated that the noise was located at the manual block valve downstream from the check valve and that it was almost unnoticed at the check valve.
o Maintenance orders were prepared to open the line and investigate the cause of the noise at the next available opportunity., Initially, it was expected that this would be at a scheduled outage in October, but, because the noise had remained constant for the prior 4 months, the work was then deferred to the start of the refueling outage, acheduled for November 30.
o Both OSRC at the Station and the Nuclear Safety Group at the GO reviewed the report of the investigation, the evaluation of safety significance and planned course of action.
When this noise appeared in 1985, it had been almost 10 years since there had-been any problems reported with the regulator 4
discharge check valves.
Personnel who had been directly involved with the earlier problems had retired or left the Station.
Anyone still at the Station who recalled the problems with the valves prior to the n ifications in 1977 might still have thought that the earlier problems had been cured by those modifications, and the inservice testing program implemented in 1978 would seem to have supported that conclusion.
On the other hand, because the unit had not operated for extended periods in 1980-1981 and in 1982-1984, the actual operating time had been less than 3 fuel cycles during that nearly decade-long period.
While the modifications may have made the valves more. resistant to damage, analysis has shown that operation at reduced power, which began in 1981, will increase the damaging effects of flow-induced vibration.
CONCLUSION Experienc.e.with the five check valves that were found failed following the November 21,.1985, event is that:
o The two 12-inch MFW pump discharge check valves and three 10-inch feedwater regulator discharge check valves have significantly different histories, and they should be discussed separately, o
Until recently, the 10-inch valves were considered by the Station to be non-safety-related.-
i
.L
,o DR. L.
T.
PAPAY February 14, 1986 o
Considerable difficulty was experienced with the 10-inch valves, until they were modified by Steam Division Maintenance in 1977.
These modifications appeared to have corrected the problem, as confirmed by subsequent trouble-free operation and extensive surveillance testing commenced in 1978.
o However, starting in 1981, the 10-inch valves would have been subjected to more severe service, since they clearly (i.e.,
by both analysis and physical evidence) are not fully open under reduced feedwater flow conditions.
Thus, the modifications made in 1977 might have fully resolved the earlier problem.
The 12-inch check valves experienced few problems, and those o
that were experienced would be considered isolated.
The bolted disc assembly of the Pacific Valves design is o
subject to failing to remain tight, leading to unscrewing, and to failure of the threads and nut. pin under vibration impact.
The replacement valve design is not subject.to~
either of these failure modes.
Assembling this experience has required more than 1,000 manhours, which were expended over the last two weeks, to search all kinds of records unrelated to the check valves themselves.
These records include Control Room logs, outage requests, and records for associated equipment such as the MFW pump and feedwater regulator valves.
If you have any questions or comments, or if you would like additional information, please let me know.
/
H ROLD B.
R HBR/jkb: 0067 cc: Kenneth P. Baskin Glenn J.
Bjorklund Harold B. Ray Site Management CDM
,