View source for IA-87-261, Submits Completed Evaluation Re Problems Encountered at Facilities W/Lpsi Pumps During Unit 1 Hot Functional Testing.Problems Resolved in Satisfactory Manner.Evaluation Completes Task Interface Agreements 83-73 & 83-13

{{Adams
| number = ML20244D282
| issue date = 03/20/1985
| title = Submits Completed Evaluation Re Problems Encountered at Facilities W/Lpsi Pumps During Unit 1 Hot Functional Testing.Problems Resolved in Satisfactory Manner.Evaluation Completes Task Interface Agreements 83-73 & 83-13
| author name = Thompson H
| author affiliation = NRC OFFICE OF NUCLEAR REACTOR REGULATION (NRR)
| addressee name = Kirsch D
| addressee affiliation = NRC OFFICE OF INSPECTION &amp; ENFORCEMENT (IE REGION V)
| docket = 05000528, 05000529, 05000530
| license number = 
| contact person = 
| case reference number = FOIA-87-261
| document report number = NUDOCS 8504120336
| package number = ML20235A611
| document type = INTERNAL OR EXTERNAL MEMORANDUM, MEMORANDUMS-CORRESPONDENCE
| page count = 15
}}

=Text=
{{#Wiki_filter:--___         - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .      _
           ..                                                                                                                 )
8  m'oIg                                UNITED STATES                    "L :m 4
[" g          g               NUCLEAR REGULATORY COMMISSION WASHINGTON D. C. 20555 r, g -
   %, +N
              /                             MAR 2 01985              i!5..: -J p" :                                  f.
Docket Nos.: 50-528, 50-529                                                                                I and 50-530                                              N I
MEMORANDUM FOR:    Dennis F. Kirsch, Actino Director                                                                  l Division of Reactor Safety and Projects                                                            !
Region V FROM:              Hugh L. Thompson, Jr., Director                                                                    ]
Division of Licensing                                                                              l Office of Nuclear Reactor Regulation

==SUBJECT:==
COMPLETION OF EVALUATION OF PROBLEMS ENC 0UNTERED AT PALO VERDE DURING HOT FUNCTIONAL TESTING IN UNIT 1 AND WITH THE LPSI PUMPS (TIA NOS. 83-73 AND 84-13)                                                                   ,
By memorandum from Thomas Bishop, dated August 4, 1983, Region V requested that                                       i NRR assume lead responsibility to assess the causes of failure and the adequacy of engineering resolution of the problens encountered with the reactor coolant                                        l pumps, thermal liners, thermowells and the CEA shroud during hot functional                                           '
testing at Palo Verde Unit 1 (TIA 83-73). By memorandum from T. W. Bishop, dated February 6, 1984, Region V also requested that NRR assume lead responsibility for the evaluation of the problems encountered with, and the modifications for, the LPSI pumps at Palo Verde (TIA 84-13).
We have evaluated the above matters as part of the operating license review for Palo Verde. This review included an evaluation of the cause of the problems for each of the five components, the modifications made to correct the problems, and the results of tests performed before and after the modifications were made.
Our completed evaluation is included in Section 14 of Supplement No. 7 to the Palo Verde SER, dated December 1984. A copy of the evaluation is enclosed.
As indicated in Supplement No. 7, the causes of the problems were identified by examinations, analyses and tests, and the modifications made to the components were shown by test to have corrected the problems. On the basis of our evaluation of all of the information provided, we concluded that the problems were resolved in a satisfactory manner.
Although the problems occurred primarily on Palo Verde Unit 1, they also relate to the components at Palo Verde Units 2 and 3. As a result, Arir w Public Service is making, or has made, the same modifications to the c wponents in Units 2 and 3. Except for WNP-3 (which is the only other CESSAP. System 80 plant), we do not find any generic implementations to the_froblems beyond the CESSAR System 80 design, c - ac .
u,u, - a. - g p y g4ow                                                                                 mm 4

i This completes our evaluation of TIA Nos. 83-73 and 84-13.          I kcutb ,]l fugh L.      mpso ,  r., Director KDivisionofLicensing N      Office of Nuclear Reactor Regulation k

==Enclosure:==

As stated                                                                       i i
l

l
, .      .                                                          )
In Supplement No. 6 to the SER, the staff stated that it was reviewing the resolution of problems encountered during' preoperational testing with the       -
reactor coolant system (RCS) pumps, low pressure safety-injection (LPSI) pumps,      j thermowelds,thermalsleeves(liners),andcontrolelementassembl9 shroud.                 l Th'e staff has now completed its evaluation of these component problems. The staff's evaluation included review of all reports submitted by the applicant on each problem (interim reports, final reports, and other supplemental docu-ments) in addition to numerous meetings with both the applicant and nuclear steam supply system (NSSS) vendor to discuss the problem causes and solutions.
The staff's findings for each component are summarized in the following discussions. 3 (1) Reactor Coolant Pump Failures During Hot Functional Testing A number of hardware failures were discovered in the reactor coolant pumps (RCPs) at PVNGS Unit 1 following the pre-core hot functional test program.
The major problem included diffuser and suction pipe retaining cap screws that were loose and/or broken, damage to the leading edge of the diffuser vanes            4 because of cavitation, and broken impeller vane segments. These RCPs were sup-        I plied by Combustion Engineering (CE), were designed by X1ein Schanzlin & Becker (KSB) of West Germany, and were manufactured and tested by CE-KSB in Newington, New Hampshire.
The failures associated with the diffuser, i.e., the bolted connections and the diffuser vane cavitation damage, were determined not to be a materials problem but a design problem.
These failures were a result of the design of the pump as it operated at maxi-        I mum or runout flow rates. At the higher flow rates, there was a flow mismatch between the impeller blade and diffuser vane, since the impeller and diffuser were sized for the normal design flow point. This mismatch was the cause of           l cavitation on the leading edge of the diffuser vanes and occurred when the localized fluid velocities were highest. The narrow gap between the diffuser and impeller vanes increased the problem since there was little room for any localized flow adjustment.
Also, as the impeller blades pass a stationary diffuser vane, hydraulic forces are imparted to the vane. The larger the gap between the passing impeller blade and the diffuser vane, the smaller are the forces which are passed. When the radial gap between impeller and diffuser is too small, a strong shock is gener-ated each time an impeller blade passes a diffuser vane inlet. These forces can be seen at the blade passing frequency which is a function of pump revolutions per minute (rpm) and the number of impeller blades. This hydraulic loading of the diffuser was the cause of the failures which occurred in the two bolted dif-fuser connections, in conjunction with relatively low capscrew pre-loading and a joint design which could contribute to relative movement.
Extensive model testing at KSB and full-scale pump testing at CE Newington have verified these design problems at pump runout condition. Operation at single-pump runout, which is approximately 142% of design flow, produced the highest pulse intensities and, therefore, the highest stresses in the working parts.
Palo Verde SSER 7                      14-2

Examination of the impeller vane fracturg surfaces indicated over* stress fail-ure by fatigue. Extensive investigations were made of the impeller castings which indicated that the three failed vanes were the thinnest of the 22 vanes examined. A finite element stress analysis was performed to betted understand the impeller failures and estabifsh a basis for ensuring that new impellers would have an adequate margin against failure. The peak stress distribution was shown to be near the leading edge of the impeller vane in the fillet area of the hub connection. This is where the cracks which led to the vane failure are located. .In addition, st min gauge instrumentation in the full-scale pump                     l test program was used to ver:'y the stress levels in the impeller vane fracture                  ;
area.
A number of design changes were made to the pumps, as discussed below, to cor-rect the problems discovered during the hot functional test program.
The radial gap between diffuser and impeller increased from 2% to 6% (material was removed from diffuser vane to accomplish this), which demonstrated a signif-icant reduction of the potential for cavitation damage on the diffuser inlet during operation at the low temperature runout flowrate condition. This change also reduced the pressure pulsations and hydraulic loading on the diffuser and, therefore, reduced the stresses in the diffuser's bolted connec-tions. In addi-tion, the diffuser's inlet vanes were re profiled.
The str.ength of the diffuser and suction pipe-to-diffuser joints was increased.
The number of bolts, length of bolts, and bolt torques at these joints were all increased. In addition, other design changes were made to increase the stiff-ness of these joints.                                                                             ;
i The impellers were replaced with impellers that had thicker vanes near th'a lead-ing edge where the failures had occurred. The trailing edge of the impeller                       i vanes were backfilled to bring the pump head curve back up to design (trailing                    i edge did not fail in hot function testing). These modifications provided a                        ~
safety margin of 1.75 for the peak stress relative to the thickest vane which previously failed.                                                   '
i A test program was carried out at the CE Newington test facility to verify the                     ;
modified design. This testing included 50 hours at design flow rate and 100 hours at runout flowrate on the original pump design to collect baseline data on the full-size hydraulic components at operating temperature and pressure.
The modified pump was then tested for 51 hours at design flowrate and 150 hours at runout flowrate to verify the modified pump hydraulics. These tests included j
(1) strain gauge measurements on diffuser bolts, (2) accelerometer data to indi-                   ;
cate vibration levels in the diffuser flange, (3) pressure pulsation data and                      i visual inspection to check for cavitation marks, contact surface wear, or move-                    3 ments, and (4) bolt torque values,                                                                 j In addition, model testing was conducted at KSB to verify that the increase in                     !
impeller-to-diffuser gap reduced the radial hydraulic forces. The model test-                       .
ing also provided fiber-optic investigation of the cavitation phenomenon to                        !
support the fact that these changes reduced the tendency for local cavitation in the diffuser. The KSB model tests and CE prototype tests were also used to verify the impeller stresses.                                                                      j Palo Verde SSER 7                       14-3           ,
i

                                                                                         .1 4
In addition, a demonstration test was conducted at PVNGS Hnit 1 to confirm the adequacy of the repairs to the react'or coolant pumps under operating conditions.
RCP 28 was torn down and inspected after completing 737 hours of ogeration (37 hours were at runout conditions). Visual inspection of the diffuser vanes and the diffuser and suction pipe bolted joints showed no evidence of cavitation or loose cap screws. The impeller was inspected and it passed the nondestructive examination (NDE) testing criteria previously established.
The visual inspection of the impeller revealed minor cavitation on the convex side of three vanes in a low stress area removed from where impeller vane fail-ures had previously occurred. The average area of the cavitation was 16 mm in diameter. Exp.erts from CE-KSB in Newington and from K58 in Frankenthal, West     i Germany, were' consulted on the finding and they agreed that the slight cavita-       I tion was acceptable. This conclusion was based on the extensive experience            ,
derived from similar testing in Fest Germany where such cavitation was found to       l be self-limiting, and because the location of the cavitation was in the area of       '
least stress and away from the area previously deemed critical.                       j Although the RCP is not a safety-related component, the staff has followed closely the applicant's evaluation of the RC.P problems which developed during hot functional testing at PVNGS Unit 1 to determine the potential impact on plant safety. The staff followed the program for determining the root cause of the deficiencies, as well as the program for verifying the modifications by analysis, model testing, prototype testing, and full-scale field testing. The staff has also reviewed the applicant's final report on this matter, submitted by {{letter dated|date=September 14, 1984|text=letter dated September 14, 1984}}, and a subsequent {{letter dated|date=September 27, 1984|text=letter dated September 27, 1984}}, which summarized the results of the inspection following the demonstra-tion test. It is the staff's opinion that the applicant's modifications to the RCP have resolved the deficiencies and that the RCP does not have any cr. edible failure mechanism which would have safety implications.
(2) Low-Pressure Safety-Injection Pumps Failure to Start The PVNGS 1-3 low pressure safety-injection (LPSI) pumps are supplied by Combustion Engineering (CE). The pumps are manufactured by Ingersoll Rand (IR) and include 500-hp Westinghouse motors. Such a problem (an LPSI pump failure to start) was discovered during the preoperational testing on PVNGS Unit 1.
Pump disassembly and inspection revealed surface damage to the mating surfaces of the impeller and pump lower case wear ring. The damage was repaired by smoothing these surfaces and the pump was then successfully retested.
During subsequent preoperational testing, additional failures to start were
,   encountered with the LPSI pumps.        The failures to start were intermittent, with one failure to start occurring after 41 successful starts and an accumu-lated run time of 66 hours.
The cause of the failure to start was hard contact between the impeller and casing ring. On the basis of a recommendation of CE and IR, the following cor-rective actions were implemented on the LPSI pumps to mitigate the effects of the contact. Because of similarities in design, the same changes were imple-mented on the containment spray (CS) pumps.
(1) The upper and lower case rings were replaced with material known for its gall-resistant properties (ARHC0 Hitonic 60).
Palo Verde SSER 7                          14-4

i                       )
l      (2) The running clearances between the jmpeller and case rings were ir. creased.
(3)  Theimpellerupperandlowerringfitareaswereserratedtopakethem less sensitive to any contact.
(4) Alignment constraints were increased to ensure centralization of the upper case ring.
However, following these changes to the pumps, additional failures to start           i occurred. As with previous failed starts, the shaft rotated slowly before trip and was free to rotate by hand thereafter. At that time it was concluded that shaft flexibility combined with transient electromagnetic starting forces of this parti ~cular 500-hp motor were responsible for the impeller contacting the wear ring with a resultant failure to start. Although contact between im-pellers and case rings during startup did not in itself cause failure to start, it was demonstrated to be a precondition of the failure-to start mechanism.
Therefore, to minimize the startup shaft and impeller deflections, a stiffer          I shaft 800-hp motor from the CS pump was installed on the LPSI pump. Measure-ments of startup deflections have shown impeller-to-case ring contact consist-ently occurs with the original 500-hp LPSI pump motors and does not occur with the stiffer shaft 800-hp CS pump motors in combination with either LPSI or CS         i I
pump impellers.
It was then determined that a 100 start test with no failures would demonstrate adequate reliability with 95% confidence for a pump / motor set. An LPSI pump using an 800-hp CS pump motor successfully completed this test. In addition,         l l
PVNGS Unit 1 LPSI pumps A and B have been started 36 times and 46 times, respectively, since completion of the 100-start test, without any difficulties.
Disassembly and inspection of these pumps during the time period of additional starts have not disclosed any abnormal wear patterns. CS pumps A and B have           !
been started 48 times and 46 times, respectively, with the same results.
After reviewing the final report for the LPSI pump failure-to-start problem and the test results, submitted by {{letter dated|date=August 9, 1984|text=letter dated August 9, 1984}}, the staff concurs with the applicant's conclusion that the safety-related LPSI and CS pumps are qualified to carry out their intended functions with the modifica-          ,
l tions described.                                                                      I l
(3) LPSI and CS Pumps Abnormal Rumbling Noises                                        !
During the performance verification testing of the modified LPSI and CS pumps,         {
a rumble-type noise was observed in the pumps and their adjacent suction piping. The rumble occurred between 2,800 and 3,400 gpm in the LPSI pumps and    ,
between 1,800 and 2,800 gpm in the CS pumps, which are below the normal flow           l ranges for these pumps. The rumble was intermittent, not periodic in character.        !
Before LPSI and CS pump modifications, the pumps had not been operated in these        ,
flow ranges for a sufficient time to determine if the rumble was present even          l before pump modifications had been made.
The rumble noise came from collapsing of bubbles in the flow stream about one
                                                                                           \
l foot below the pump casing in the intake pipe, Aural observations of the               !
intake piping at several locations disclosed that strong turbulence develops in        i the flow aperiodically. The bends, tees, and reducers in the system are suf-           l ficient to generate random, large-scale turbulence. The cavitation conditions Palo Verde SSER 7                       14-5 l
l

           -                                                     i then develop intermittently when the swirl, associated with a burst of turbu-lence, interacts with the prerotation induced in the intake pipe when operating     i the pump at partial flow conditions.
I Tests conducted on similar pumps have demonstrated that backflow from the impeller can induce prerotation in certain partial flow ranges. Accelerometer       i data confirmed the propagation downstream of flow disturbances at the acoustic      l wave speed which coincided with the noise source. In addition, changes in pipe internal configuration upstream of the intoke (addition of strainer in the          l eccentric spoolpiece) shifted the frequency at which rumble occurred.
i The upper tinte limit for which conditions (flowrates) at which rumble could         !
occur in the LPSI pump is 4 hours. The CS pumps will not be operated at all         j in the flowrate range for which rumble occurs. LPSI pump 1B was run in its          l rumble range during tests for a duration of about 2 hours. Post-test inspec-tions revealed no pump degradation. Also, IR has confirmed that    operation in the rumble range for up to 4 hours would not cause pump damage.
The applicant has therefore concluded that both the LPSI and CS pump system do not represent a safety concern if left uncorrected and would not adversely           ;
affect the capability to safely shut down the reactor. However, LPSI pump           I operating procedures are being revised to incorporate a warning not to operate       l in the 2,500 to 3,500 gpm flow range during the shutdown cooling mode of            l operation.
After reviewing the final report and test results for the LPSI and CS pump           .
rumble condition, submitted by {{letter dated|date=September 26, 1984|text=letter dated September 26, 1984}}, the staff           l concurs with the applicant's conclusion that the safety-related LPSI and CS          l pumps are qualified to carry out their intended safety functions without             i requiring any modifications regarding operation in the rumble flowrate ranges.       j (4) Thermowells Hot functional testing (HFT) at PVNGS Unit I was initiated in early May 1983.        l The initial indication of resistance temperature detector (RTD) and related          '
equipment problems developed at the site when the first of five RTDs failed in the electrically open position on May 31, 1983, during the HFT. The RTD senses reactor coolant temperatures at various locations in the primary loop. The thermowell forms a pocket for mounting the RTD by penetrating the reactor coolant system (RCS) piping and providing a thin-wall membrane which isolates primary system pressure. HFT was about three quarters complete on June 17, 1983, when a leak was detected in the thermowell corresponding to the first RTD that failed electrically. On June 21, 1983, a leak developed in the thermo-well associated with a second RTD that failed. APS and CE site personnel analyzed the pattern that had been established, i.e., the failure of an RTD and subsequent failure of the associated thermowell, and proceeded to plug those thermowells which contained failed RTDs.
When the loop 2A reactor coolant pump (RCP) was disassembled for its planned inspection following HFT, the cold-leg thermowells in loop 2A were inspected through the RCP casing with the pump diffuser in place. No thermowell failure was detected and further HFT was performed. Structural vibration data for the thermowells was obtained during this testing by placing an accelerometer in one Palo Verde SSER 7                       14-6

l l
of the thermowells. Subsequent to these tests, inspection of the thermowells       l from the inside of the RCS piping during the week of July 18, 1983, showed         '
damage to several cold-leg thermowells. Some cold-leg thermowellsgwere broken flush with the inside of the RCS pipe; one was bent but intact; and one was broken both M the intersection between the large section at the top of thermo-     l well and at the lower end adjacent to the inside wall of the pipe. Another thermowell was broken at the top and had fallen into the flow stream of the RCS cold leg. Other thermowells showed no visible damage. A total of five cold-leg thermowells were found to have failed. Initial inspection of the hot-leg thermowells did not show any visible damage, except about half of them were slightly bent in the direction of the reactor vessel (against the flow).           !
Both visual and metallurgical examinations were performed on the damaged           l thermowells. The visual examination included wear measurements which showed that the most significant wear was experienced in the RCS cold legs, which can have higher than normal flow. The high-flow conditions were experienced in various RCS cold legs during HFT when only one of the two reactor coolant pumps was operated, inducing flow in a particular steam generator. The majority of the thermowells that failed (3 out of 5) were located in the cold leg that had the highest number of hours in this high-flow mode of operation.                   .
l The wear measurement and damage correlation also showed that thermowells at a       l particular location in the RCP cold legs were the most susceptible to both wear    I and damage. On each cold leg, three thermowells are installed approximately 30 in. from the pump. They are oriented at 10, 12, and 2 o' clock when viewed from the pump in the direction of flow. The 10 o' clock position thermowells received the worst damage in 3 out of 4 loops. This position is almost in a direct line with the flow axis of the RCP diffuser vanes.
There is no physical evidence that a broken impeller part from the reactor coolant pump impacted any of the thermowells on loop 1B. On loop 2A, it appears Thermowell No.125 was struck very early in the HFT period because          ;
little wear took place before it was bent at about 45' as a result of impact.      1 Thermowell No. 122CA on the same loop (2A) was also struck but only after a        !
considerable amount of wear occurred. This thermowell also fractured at the        {
top.
1 A visual examination of the wear surfaces on the dowstream side of the thermo-wells classified the wear as adhesive wear. This wear is typical of that pro-      3 duced by oscillatory motion of loaded contact surfaces.
l A metallurgical examination was performed on the five failed RTD thermowells.      {
The results indicated that the chemical and mechanical properties and the          !
microstructure were within the normal limits. There were no indications of         I pre-existing flaws on the fracture surfaces.      The fracture surfaces exhibited  i relatively large areas of fatigue cracks. The cracks indicate high-cycle (low-     j stress) fatigue as the failure mechanism. Possible crack initiation points were    i identified on the outside of the thermowell tubular sections at approximately      !
90 to the flow direction.      Portions of the fracture surface were smeared       j because of relative motion of the two surfaces. It was concluded that the most     l likely excitation mechanism to cause this type of failure would be vortex          l shedding. Vortex shedding results when flow across a tube produces a series of  :
vortices in the downstream wake formed as the flow separates alternately from      i Palo Verde SSER 7                       14-7 I                                                                                   J

em
           .                                                      1
.                                                                                               1 I
i the opposite sides of the tube. Thi.s alternating shedding of vortices produces              l alternating forces which occur more frequently as the flow increases.
Calculations have shown that for rarmal operating flow rates, the kortex shed-ding frequencies for the cold-leg thermowells would be adequately separated from the predicted natural frequency. For the higher flow conditions that existed during some portions of the hot functional testing, the vortex shedding frequency can be analytically shown to be close to the thermowell natural fre-quency, and thus could have stimulated the thermowell at its natural frequency.
To determine if the vortex shedding mechanism is responsible for thermowell damage, tests were run at the Combustion Engineering TF-2 flow loop test facil-ity in Windsor, Connecticut. Testing of the System 80 thermowell/ nozzle pro-duced a wear pattern similar to that Observed after HFT of PVNGS Unit 1. As a                 j result of the damage and the postulated .Silure mechanism, CE initiated a pro-gram to redesign the thermowell in order to increase its strength and stiffness to raise its natural frequency.
The redesign of the thermowell is based on maintal.'ing the original interfaces, design parameters, and thermal response times for the RTD instrument. In addi-tion, it was desired to minimize flow-induced excitation.       Four major design objectives were established:
(1) Increase the natural frequency of the thermowell to prevent resonance with vortex-shedding frequencies.
(2) Eliminate the clearance at the support area between the thermowell and nozzle to eliminate relative motion that could cause wear.
(3) Reduce stress level to eliminate the possibility of high-cycle fatigue.
(4) Provide a flow profile that would minimize vortex-induced loading.
A structural analysis was performed for pressure, thermal, seismic, and mechan-ical loadings for the redesigned thermowell. The thermowell was designed to the requirements of the ASME Boiler and Pressure Vessel Code Section III for Class 1 componen*'
The redesigned thermowell was tested in the CE-Windsor TF-2 flow loop to observe the effects of vortex shedding without the influence of the reactor coolant pumps. The redesigned thermowell was also installed in the CE-KSB pump test loop near the RCP outlet, similar to its actual arrangement in the reactor coolant system. PVNGS Unit 1 flow velocities and the test flow velocities were compared and it was concluded that the design calculation assumptions were adequately conservative.
In addition, shaker-table tests were conducted to compare the natural frequency of the original design with that of the redesign. The results of these tests indicated that the natural frequency of the new design is higher than that of the original design by a factor of 2.
The redesigned thermowell was then tested during the demonstration test at PVNGS Unit 1. The purpose for testing the thermowell was to verify that the thermowell response was consistent with that observed from other tests and by Palo Verde SSER 7                       14-8

analysis. The test duration was 700 hours. Test results verified that all design objectives were accomplished.. Specifically, the response of the redesigned thermowell to vortex shedding was substantially reduced      Visual inspection of the thermowells after completion of the demonstratiob tests showed no damage or wear.
The staff has reviewed this matter, including the applicant's final report regarding resolution of the issue, submitted by {{letter dated|date=September 14, 1984|text=letter dated September 14, 1984}}. After evaluating the analytical results and test data submitted by the applicant on this subject, the staff concurs that thermowell failures were caused by the resonance of vortex shedding frequencies and the thermowell             j natural frequ.ency which resulted in wear and high-cycle fatigue. The staff concludes that analyses conducted by the applicant, supplemented by test data from the CE-Windsor TF-2 flow loop, the CE-KSB pump test loop and the full-           i scale demonstration tests satisfactorily demonstrated that the new thermowell design is structurally adequate.
l (5) Thermal Liners During the post-HFT inspection on July 19, 1983, the reactor coolant pump (RCP) 1A and 18 discharge piping was entered to look at the thermowells which failed during the HFT. It was noticed that the thermal liner in the safety-injection nozzle for the IB pipe was protruding into the pipe about one-half inch.      Also, it was observed that the thermal liner was missing from the safety-injection nozzle in the 1A pipe and there were gouges in the cladding on safety-injection nozzle 1A near the nozzle-to pipe juncture where the positioning pads were located. The missing liner was found in the reactor vessel below the inlet nozzle through which it had passed and wedged between the reactor vessel and the outside of the flow skirt. All other nozzles with thermal liners in the         i RCS piping were examined, and the liners were found to be in place.                    l The applicant recovered thermal liner 1A from the reactor vessel. Inspections and examinations showed that the nozzle groove was correctly machined, and the liner was explanded (expanded by explosion forming) into place properly, but the liner had vibrated and had worn the nozzle cladding so it became loose and eventually dislodged from the nozzle. The safety-injection nozzle is located downstream of the pump and upstream of the reactor vessel. The applicant eval-uated the potential for blocking core flow and concluded that the dislodged liner would not lead to flow blockage. The liner would be prevented from enter-ing the core region by the reactor flow skirt and would remain trapped between the flow baffle and the reactor vessel shell as found in PVNGS Unit 1. However, the liners were originally installed only as additional assurance of adequate protection of the nozzle. Because of this and to avoid loosening and possible failure of the three remaining liners, the applicant decided to remove all thermal liners from the safety-injection nozzle areas. Any damage done to the nozzle cladding was repaired and operational suitability was verified by non-destructive examination. The explansion ridge in the cladding was also removed and the surface was machined smooth and examined. No base metal was exposed.
To demonstrate that the above modification is acceptable, the applicant reviewed and reexamined the usage factors for the safety-injection nozzles based on all design transient cycles. The maximum cumulative usage factor in the part of the nozzle that is protected by the liner when it is in place is calculated to be 0,094. The usage factor at this location without the liner is 0.34. The Palo Verde SSER 7                       14-9

l
        ,                                                            )
cumulative usage factor is calculated to,be 0.60 in its "as is" configuration in which a stress concentration factor is used. If this surface is machined smooth so that a stress concentration factor would not be present,g which is the case for the modification described above, the usage factor at thit location would be 0.16. Thus, the largest usage factor in the area that was behind the liner will be 0.34 when the liner is not present. The usage factor in the safe-end portion of the nozzle, which is not protected by the thermal liner, is
: 0. 6. Therefore, the absence of the liner will not change the operating capabil-ity of the nozzle.
The applicant has removed all thermal liners from the safety-injection nozzle areas together with the explansion ridges and repaired all the damages. The staff has rev'iewed this matter, including the applicant's report regarding resolution of the issue submitted by {{letter dated|date=December 30, 1983|text=letter dated December 30, 1983}}. Since the cumulative usage factor in the area that was behind the liner is a maximum of 0.34 compared with the usage factor of 0.6 at the safe-end portion of the nozzle, the staff concludes that the modification eliminated the potential problem and will not affect the operability of the nozzles. The staff agrees to this modification and finds it acceptable.                                       '
(6) Control Element Assembly Shroud (CEA Shroud)
Inspection of the PVNGS Unit I reactor internals subsequent to HFT in July 1983 revealed damage to the control element assembly shroud (CEA shroud).       The CEA shroud is part of the upper guide structure (UGS) assembly. It consists of an array of vertical round tubes (9-in. OD) which are arranged in a square grid          !
pattern with 16-in. pitch. The tubes are joined by welding vertical plates             1 called webs between adjacent tubes, as shown on Figure 14.1. Tubes and webs             '
are made from 3/16-in. type 304 stainless steel. The purpose of the CEA shroud is to provide separation of the CEAs. The CEA shroud is mounted on eight pads on the UGS base plate and is held in position by eight tie rods which are              ]
threaded into the UGS base plate at their lower end. At their upper end, the            l pretensioned tie rods are held by nuts which bear on eight plugs in the tops of         1 eight of the CEA shroud tubes. Guides for the 4-finger CEA extension shafts are attached to the top of the tubes, and guides for the 12-finger CEA extension shafts are attached to the webs. These guides serve the purpose of aligning CEA extension shafts for entry into the closure head nozzles during closure head installation and into the internals lift rig during attachment.
The damage, revealed by visual and dye penetrant examination consisted of the following:
(1) A total of 13 cracks in eleven 4-finger CEA shroud tubes. In most instances, these cracks start in the welds at the attachment of the 4-finger CEA guides to the shroud tubes.
(2) Two cracks involving the welds at the attachment of the 12-finger CEA extension shaft guides to the webs.
(3) Three cracks involving the welds between 4-finger CEA shroud tubes and webs: two at the top of the shroud and one at the bottom.
(4) One crack in the base metal of a web.
Palo Verde SSER 7                         14-10

                                                                   )
i
      .                                                                              i
    .                                                                                l (5) Three wear marks on the shroud 'at the.45" location.                         i (6) One ductile break, one-half-inch long, located in a web at thb bottom.
A' metallurgical program was established to identify the nature of the failures.
Samples of the shroud were removed and examined. Metallography and chemistry  l confirmed that the shroud material was as specified. Fractography of the         j fractured surfaces showed most of the failures occurred by high-cycle fatigue-   i i.e., by induced cyclic stresses of a magnitude at or near the endurance limit of the material. Cracking in some of the welds was identified as transgranular stress corrosion cracking (TGSCC) which was caused by entrapped slag from the-shielded metal arc welding (SMAW) process which was used in fabricating the CEA shroud assembly. The fatigue cracking and TGSCC were determined to be unrelated events except for one location. At the bottom of the shroud at a tube-to-web joint on tube 13, a fatigue crack was identified, but it occurred      ,
as a result of TGSCC propagation.                                                 l A series of hydraulic and mechanical vibration tests was performed to identify    {
potential forcing functions which might induce shroud vibrations and to charac-   '
terize the modes of vibration of the CEA shroud assembly and of individual CEA    l shroud tubes.                                                                     l l
Analyses were conducted to investigate potential causes of the failures in the     '
CEA shroud by examining the structural response to the loadings experienced during the comprehensive vibration assessment program (CVAP) which was com-pleted in July 1983. The primary objective of the analyses was to identify potential forcing functions and consequent modes of vibration during normal        l operation which could have caused the same kind of failures.                      I The evaluation of the failure modes utilizing CVAP test data, structural analy-ses, and experimental test measurements on both single-shroud tubes and on the entire CEA shroud assembly initially identified four potential failure mechan-isms for the original shroud design; of the four, only two are considered probable.      One mechanism is the lateral response of the CEA shroud assembly to vibratory excitation of the upper guide structure support plate, and a second mechanism is the higher frequency shell response of the individual shroud tubes.
Of these two, only the first was determined to be significant by analysis and was shown to be a probable cause of the failures.
Specifically, the applicant concluded that the fatigue failures of the original structure were primarily caused by low-frequency response of the assembly to excitations induced by adjacent structures (CEA tube bank) with secondary con-tributing stresses from shell mode responses due to pump pressure pulsations.
On the basis of the above evaluations, the applicant decided to modify the Palo Verde CEA shrouds by removing 3 in from the top of the CEA shrouds and also removing all of the 4-finger and 12-finger CEA guides. This eliminates the potential resonance failure caused by vibration of the CEA guides. It also eliminates the high stress concentration at the top of the tubes and thereby reduces the local stresses induced by global shroud vibration. In addition, it effectively eliminates all of the original locations of crack initiation.      The guides as originally designed had no function during normal operation. They served the purpose of aligning CEA extension shafts for entry into the closure head nozzles during closure head installation and into the internals lift rig during attachment. With the modifiod design, this function is provided by a Palo Verde SSER 7                        14-11

m
* 1
        ~
                    .                                                    >                       \
separate tool which is not a permanent part of the vessel or the internals.
This tool is utilized only during refueling operations.
Toeliminatethepossibilityoftransgranularstresscorrosioncrabking,all fillet, double fillet, or partial penetration welds which had been made pre-viously with shielded metal arc process were mechanically removed and replaced by the gas tungsten arc proce:s.                                                        l The second modification was the addition of snubbers (keyways) which limit the          .
lateral displacement of the CEA shroud in the global modes of vibration.               I Snubbers are located on the shroud at the upper guide structure flange eleva-           l tion and transmit the loading to the UGS flange. This also raises the natural          '
frequencies o~f the dominant global vibration modes of the shroud relative to the UGS assembly.
To characterize the vibration behavior of a single CEA shroud tube after it had been modiffer as described above, a mechanical excitation test was performed in air.
Results indicated that the maximum strain amplitude existed at the least critical tube reghn for the resonance frequency. This location is far away             I from the web welownts where stress risers would exist. Analytical models were confirmed by using the detailed model deflections and strain patterns obtained from these experiments.
For a final assessn.ent of their design adequacy, the modified CEA shrouds were includad as'a part of the demonstration test which was conducted by the appli-cant at PVNGS Unit 1 in July 1984. The test conditions were representative of        -
those used during the hot functional test and the system was appropriately instrumented to check out the structural and hydraulic performance. Following completion of the test, the reactor vessel head was removed and a reactor cool-ant pump was disassembled to be visually inspected.
Evaluation of the comparisons of analytical predictions and demonstration test measurements led the applicant to the conclusion that the design adequacy of the modified CEA shroud is acceptable for long-term operation.
This conclusion is based upon the following results:
(1) All design limits of Section III of the ASME Code have been met by means of analysis for normal operation and for seismic and LOCA loads.
(2) The measured response frequencies of the CEA shroud assembly were as                l predicted. Response strains from assembly motion were lower than expected.      l The shell mode response was shown to be small and well under the accept-        ;
ance criteria.
(3) The acceptance criterion based upon ASME Code fatigue limits, was at no             ,
time exceeded during the demonstration test.                                    J (4) Inspection of the shroud assembly was performed after acquiring a minimum of 107 cycles of vibration. No indications of failure or abnormal wear were found.
Palo Verde SSER 7                         14-12

4 (5) Measured responses in the UGS tube hank region agreed very well with CVAP data. It can be concluded that the structural modifications do not affect-the UGS responses and flow conditions.
g (6) The mechanical excitation tests, although not representative of the load-ing in the reactor, produced failure in the original tube with the 4-finger guides but not in the modified tube. This indicates that the stress levels in the modified tube are reduced significantly for the same levels of excitation.
On the. basis of a review of the applicant's evaluation of the damaged CEA shrouds, including the final report submitted by letter dated September 14,               8 1984, the sta'ff agrees with the appifcant's assessment that two separate mechanisms, high-cycle fatigue and transgranular stress corrosion cracking, contributed to the observed cracking. The staff-further concurs that the fatigue failures were primarily caused by low-frequency response of the CEA to excitations induced by adjacent structures.                                               ;
The structures were modified to eliminate the potential resonance failure caused by vibration of the CEA 4-finger and 12-finger guides and to eliminate             l the locations of the original failures. In addition, to eliminate the possi-              j bility of future transgranular stress corrosion cracking, the shielded metal arc weld process was not used in the modified design for fillet, double fillet, or part.ial penetration. The staff also reviewed fabrication documentation for the modified PVNGS Unit 1 shroud to ensure that such welding was not used.
On the basis of a review of the applicant's analyses, in conjunction with the hydraulic tests, mechanical vibration tests, and full-scale demonstration' test at PVNGS Unit 1, the staff concludes that the original causes of the cracks have been effectively removed and that the modified CEA shroud design is acceptable for long-term operation.
i e
i l
i l
Palo Verde SSER 7                        14-13                                            l l

i 12-FINGER EXTENSION SHAFT GUIDE I
1 4-FINGER EXTENSION SHAFT GUIDE i
BACKING PLATE   ,
I           '
s     ,
                                                                                                                            )
                                                                                                                            ~
i
                                                                                        =*                      l l
                                                                              "                        .      k
                                                                     ,,                                N        /
N,                                  ,
k d
SHROUD WEB' mM    .
SHROUD TUBEi       -
        . . .              . . . .                -.            L.                  .            . . .
Figure 14.1 Control element assembly extension shaft guides Palo Verde SSER 7                                      14-14
                                                                                                                           ,}}