IR 05000361/1998019

Insp Repts 50-361/98-19 & 50-362/98-19 on 981112-990102.No Violations Noted.Major Areas Inspected:Operations,Maint, Engineering & Plant Support

ML20202F967
Person / Time
Site:	San Onofre
Issue date:	01/28/1999
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20202F959	List: IR 05000361/1998019 ML20202F954
References
50-361-98-19, 50-362-98-19, NUDOCS 9902040215
Download: ML20202F967 (25)

Text

.

.

ENCLOSURE U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

Docket Nos.: 50-361 50-362 License Nos.: NPF-10 NPF-15 Report No.: 50-361/98-19 50-362/98-19 Licensee: Southern California Edison C Facility: San Onofre Nuclear Generating Station, Units 2 and 3 Location: 5000 S. Pacific Coast Hw San Clemente, Califomia Dates: November 12,1998, through January 2,1999 Inspectors: J. A. Sloan, Senior Resident inspector J. G. Kramer, Resident inspector J. J. Russell, Resident inspector S. L. McCrory, Operator Licensing Examiner J. E. Whittemore, Senior Reactor inspector Other personnel: M. B. Fields, Project Manager, OHice of Nuclear Reactor Regulation Approved By: L. J. Smith, Acting Chief, Project Branch E Division of Reactor Projects i

ATTACHMENT: Supplemantal Information 9902040215 990128 PDR ADOCK 05000361 G pm

_ __ _ _ _ __- _ _ _ , . _ - _ _ _ . _ _ _ _ _ _ _ _ _ _

.

. i EXECUTIVE SUMMARY San Onofre Nuclear Generating Station, Units 2 and 3 NRC inspection Report 50-361/98-19; 50-362/96-19 This routine announced inspection reviewed aspects of plant operation, maintenance, engineering, and plant support. The report covers a 7-week period of resident inspection and also documents inspection activity by a regional engineering inspector and a regional operator licensing examiner and a technical evaluation performed by the Office of Nuclcar Reactor Regulatio Operations

• Operators conducted two outstanding prejob briefings. Operators conducted a thorough and intrusive prejob briefing for the performance of a special response time test of the reactor trip K-relays. Operations management oversight of this issue was excellent and !

demonstrated a focus on reactor safety. The control operator's questioning attitude '

during the prejob briefing demonstrated outstanding attention to detail. In this instance, a recently implemented initiative to reduce errors was effective (Sections O and O1.3). l l

Maintenance i i

• The postmaintenance and inservice testing performed by licensee personnel was l effective and successfully validated that the Unit 3 swing component cooling water pump j would perform its required safety function (Section M1.3). I l

A miscommunication between Instrumentation and Control (l&C) technicians and operators resulted in an excore nuclear instrument channel being returned to an operable status without the intended retests being completed. For the. work uctually performed, the retest that was conducted was adequate to demonstrate operability; however, the retest was not adequate for all the work that was authorized. Operators did not know that the channel had not been intended to be returned to service because the work was incomplete (Section M1.4).

• The licensee missed opportunities to identify equipment deficiencies. In one case a wear condition on an emergency diesel generator (EDG) control air system occurred !

that was nearly identical to a previously identified condition. In another case, several

'

fasteners were discovered to have been missing from two low pressure safety injection pump motor coolers for 5 years. The licensee was responsive and thorough in resolving issues associated with the deficiencies, which did not affect operability (Section M2.1). ;

i

• A noncited violation of 10 CFR Part 50, Appendix B, Criterion XI," Test Control," I consistent with Section Vll.B.1 of the Enforcement Policy, resulted from the licensee's identification that surveillance testing of charging pump flow was inadequate to demonstrate operability. Specifically, the total loop uncertainty for the permanently installed instruments had the potential for actual flow to be less than the acceptance I

i

- _ _ ___ _

.

.

-2-criteria of Technical Specification Surveillance Requirement 3.5.2.6. Upon identification the licensee verified that actual flow met the acceptance criteria by using measuring and test instrumentation with the appropriate accuracy (Section M8.2).

Enaineerina l

• The independe.nt Safety Engineering Group performed a rigorous and thorough root cause evaluation of the failure of several resistance temperature detectors, which demonstrated effective use of advanced technology. The recommendations of the root cause evaluation were corroborated by the vendor (Section E2.1).

• The NRC staff reviewed the inspection techniques and results, the analyWal I assumptior.s and methodology, and the acceptance criteria used by the licensee to evaluate the impact of the degraded eggerate supports discovered in the San Onofre Unit 3 steam generators. The staff concluded that the amount of degradation observed in the eggerate supports would not prevent the Units 2 and 3 steam generators from performing their intended safety functions (Section E2.2).

• A nonci'ed violation of licensee procedures and Technical Specification 5.5.1.1.a was identified, consistent with Section Vll.B.1 of the Enforcement Policy. Spacifically, a technician failed to properly verify that one core operating limit supervisory system constant was consistent with the log of approved constants (Section E4.1).

l

• The Nuclear Oversight and Nuclear Fuels groups had effectively interacted with the nuclear fuel vendor to assure that fuel scheduled for loading into the facility reactor cores met the design and functional requirements. The action taken to rescind the stop work order that was issued to the vendor on July 7,1998, was appropriate (Section E7.1).

Plant SuoDort

• Scaffoding, prestaged equipment, and barriers were appropriately controlled prior to the Unit 2 outage to prevent inadvertent impact on the operability of safety equipment (Section O2.1).

l

1

- - -- . . - . .- - ~ . - - . - - . = . - _ - . . . - ~ ~ _ ~ _ - - _ - - -

,

,

'

j

<

! Report Deta!!s

.

Summary of Plant Status Both units operated at 100 percent power at the beginning of this inspection period.

i

'

On November 21,1998, Unit 2 reduced power to 90 percent to perform a heat treatment of the

<

circulating water system and returned to 100 percent later that same day. On Deccmber 17, a reactor power coastdown was initiated in preparation for the Unit 2 Cycle 10 refueling outag *

The unit was shut down at 11

22 a.m. (PST) on January 2,1999, to commence the outage.

4 The unit ended the inspection period operating in Mode 4 with a reactor coolant system cooldown in progress.

}

,

l Unit 3 operated at essentially 100 percent power throughout this inspection period, except that

power was reduced to 97 percent to perform a heat treatment of the circulating water system on

!

December 31,1998, and returned to 100 percent the same day.

!

'

l. Operations 01 Conduct of Operations O General Comments (71707)

! l The inspectors observed routine and nonroutine operational activities throughout this ,

inspection period. Some of the activities observed included:

• Running Train A EDG 3G002 for monthly surveillance (Unit 3)

!

• Starting Train B Emergency Chiller 2/3ME335 for testing (Units 2 and 3)

Shift turnover (Units 2 and 3)

= Placing swing Component Cooling Water Pump 2P025 and Train B Saltwater Cooling Pump 2P114 in standby (Unit 2)

• Diluting the reactor coolant system (Unit 2)

= Deborating the reactor coolant system (Unit 2)

= Reactor power coastdown (Unit 2)

• Starting diesel-driven Firewater Pump P220 for testing (Units 2 and 3)

= Prejob briefing for performance of a special response time test of Reactor Trip Relays K-1 through K-4 (Unit 3)

• Initiation of borating the reactor coolant system to cold-shutdown boron concentration (Unit 2)

- - _ - - .

.

.

-2-

Initiation of reactor coolant system cooldown to Mode 5 from Mode 3 (Unit 2)

Operators were thorough and methodical in preparing for and conducting routine and nonroutine evolutions. Close management and supervisory oversight of operational activities were evident. Procedure use and operator communications were consistent with written performance expectations. Specific comments on activities are discussed l

belo .2 Preiob Briefino for Special Resoonse Time Test of Reactor Trio Relavs - Unit 3

) Inspection Scope (71707)

.

On December 29,1998, the inspectors observed operators conduct a prejob briefing for l performance of a special response time test of Reactor Trip Relays K-1 through K-4 in )

Unit I l Observations and Findinos The special test was planned as the result of a 10 CFR Part 21 report of a problem at the Waterford 3 reactor facility, in which normally energized MDR 170-1 relays failed in the energized position. The failure of the K-1 through K-4 relays could result in a failure of the plant protection system to trip the reactor. The purpose of the test was to demonstrate that the relay actuation time was less than 20 mse To accomplish the test, l&C technicians planned to use portions of a logic matrix test procedure. In addition, they planned to connect a recorder across a normally closed secondary contact for the K relays. This additional action was directed by the maintenance orde The control operator and supervising control operator carefully reviewed the test and interviewed the I&C technicians regarding the specific potential impact on control room indications and plant operations. The operators were generally satisfied with the test controls. However, the Operations assistant plant superintendent, who observed portions of the briefing, questioned whether the engineering review of the test specifically addressed the K-relay operability implications of having the recorder attached. The operators determined that the maintenance order did not address this issue. Discussions with the I&C technicians provided some assurance that there were no operability issues. Nevertheless, the operators requested a specific opinion from Station Technical before allowing the test to commence. The operators deferred the test until the Station Technical engineers provided an assessment that demonstrated operability of the K-relays. Upon receipt of the assessment the operators authorized performance of the tes The test demonstrated that the K-relays performed satisfactoril .

.

-3- Conclusions Operators conducted a thorough and intrusive prejob briefing for the performance of a special response time test of the reactor trip K-relays. Operations management oversight of this issue was excellent and demonstrated a focus on reactor safet .3 Preiob Briefina for Subaroup Relav Test - Units 2 and 3 insoection Scope (61726 and 71707)

On December 1,1998, the inspectors observed operators conducting a prejob briefing for a subgroup relay test for Relay K408A, which is associated with the safety injection actuation signal to close Valve 2HV5803, containment normal sump to radwaste isolatio Observations and Findinas Operators conducted a thorough briefing of the planned test using Procedure S023-7-3.43.18, " Subgroup Relay Testing K408A and K210A," Revision During the briefing, the control operator determined that the recently revised test procedure did not test the function of a manual actuation inhibit contac Operators stopped the prejob briefing, consistent with a program Operations has recently implemented to reduce human errors, in order to resolve the question of whether the manual actuation inhibit contact was required to be tested by the surveillance procedure. Operators initiated Action Request (AR) 981200050 to track resolution of the issu The licensee noted that similar contacts were tested in other subgroup relay test procedures. However, the licensee determined that the manual actuation inhibit function was not a safety function and therefore the testing was not required. The licensee concluded that the test procedure that was being briefed adequately tested all the required safety function C_onclusions The control operator's questioning attitude during the prejob briefing demonstrated outstanding attention to detail. In this instance, a recently implemented initiative to reduce errors was effectiv .  :

i

.

-4- ,

I O2 Operational Status of Facilities and Equipment l O2.1 Preouta_ge Activities - Units 2 and 3 Inspection Scope (71707) l The inspectors walked down most accessible portions of Units 2 and 3 prior to the Unit 2 outage to assess the extent of the impact of scaffolding and equipment prestaging on operable safety system Observations and Findinas Scaffolding and prestaged equipment had been placed so that operable equipment was ,

not impacted. Appropriate seismic restraints were in place on the scaffolds assessed, l consistent with procedures. Barriers, such as watertight doors, were appropriately l controlle l On December 8,1998, the inspectors observed an unsecured step ladder erected in a l valve and piping room in the Unit 3 tendon gallery. The piping and valves were safety-related; however, the ladder was near large-diameter safety injection system i

piping and would not have damaged the piping during a seismic event. Upon notification, operators promptly secured the ladder. The unsecured ladder failed to comply with the requirements of Maintenance Procedure SO123-1-20," Seismic  !

Controls," Revision 5. This failure constitutes a violation of minor significance and is not subject to formal enforcement actio On December 14,1998, the inspectors observed scaffolding erected around Unit 3 1 Atmospheric Dump Valve 3HV8419. The inspectors reviewed work records and f determined that the scaffolding had been erected in August 1998 to support work to i correct minor leakage from the valve. The maintenance activity had been unsuccessful and the work had been replanned for the Unit 3 refueling outage in March 1999. The Maintenance manager indicated that the scaffolding had been periodically inspected, as require Conclusions Scaffolding, prestaged equipment, and barriers were appropriately controlled prior to the Unit 2 outage to prevent inadvertent impact on the operability of safety equipmen Miscellaneous Operations issues (92901)

0 (Closed) Violation 50-361: 362/97020-01: failure to provide special corrective lenses for use in a self-contained breathing apparatus to a senior licensed operator whose license required corrective lenses while performing licensed dutie .

!

l

. .~ . . -

- . ..

.. - . - -

.

.

.

'

-5-

'

The inspectors verified that the corrective actions described in the response letter dated November 17,1997, and AR 970900993 were implemented. No similar problems were identifie . Maintenance M1 Conduct of Maintenance M1.1 General Comments Inspection Scope (62707)

!

The inspectors observed all or portions of the following work activities:

• - Troubleshooting of positioner for Valve 2LV0201 A, usown backpressure control (Unit 2)

<

• Postmaintenance test of Train B Emergency Chiller 2/3ME335 (Units 2 and 3)

• Radiation Monitor 2RT7807-2 blower inspection and lubrication (Unit 2) Observations and Findinas The inspectors found the work performed under these activities to be thorough. All work observed was performed with the work package present and in active use. Technicians were knowledgeable and professional. The inspectors frequently observed supervisors i and system engineers monitoring job progress, and quality control personnel were present whenever required by procedure. When applicable, appropriate radiation controls were in plac In addition, see the specific discussions of maintenance observed under Section M1.3, i belo M1.2 General Comments on Surveillance Activities Inspection Scoce (61726)

The inspectors observed all or portions of the following surveillance activities:

• Excore Train A 2-year surveillance (Unit 2)

• Subgroup Relays K408A and K210 test (Unit 2)

• Train A EDG 2G002 monthly test (Unit 2)

• Atmospheric dump valve quarterly test (Unit 2)

In addition, see the specific discussions of surveillance activities observed under Sections 01.3, above, and M1.4, belo .

-6- Observations and Findinas The inspectors found all surveillarces performed under these activities to be thoroug All surveillances observed were performed with the work package present and in active use. Technicians were knowledgeable and professional. The inspectors frequently observed supervisors and system engineers monitoring job progress, and quality control personnel were present whenever required by procedure. When applicable, appropriate radiation controls were in plac M1.3 Testina of Swina Comoonent Coolina Water Pumo 3MP025 - Unit 3 Inspection Scope (62707)

The inspectors obsewed portions of the following work activities rebted to the swing component cooling water pump: (1) postmaintenance operability testing of swing Component Cooling Water Pump 3MP025 following replacement of the pump mechanical seal and (2) inservice testing of Component Cooling Water Pump 3MP025, Observations and Findinas The component cooling water pump postmaintenance testing and inservice testing were conducted in an integrated manner. However, a complete and separate prejob briefing was conducted for each evolution. The postmaintenance test was conducted to validate successful performance of the replaced pump mechanical shaft seal. The seal had been replaced earlier but failed immediately after initial startup. Engineering personnel identified a problem with warehouse spares, entered the deficiency into the corrective action program, and obtained the appropriate replacement sea The inspectors observed the prejob briefing conducted by the Unit 3 control operator for the initial test of the pump. The test procedure and appropriate station policies, practices, and procedures were referenced and discussed with the involved test personnel from Maintenance, Station Technical, and the Operations Test Group. The briefing iricluded a discussion of individual responsibilities during the test. In addition, test personnel discussed the expected pump performance and identified the immediate and followup actions in the event of unexpected equipment performance. The performance of the postmaintenance test of the pump was successful, and test personne: performed effectivel A successful inservice test was conducted following the postmaintenance test in i accordance with Procedure SO23-3-3.60.3," Component Cooling Water i Pump 2(3)MP-025," Revision 1. The test data was collected electronically and transferred to the Vision program for performance analysis. The inspectors reviewed the procedure and "scussed the interpretation of the test data in Inservice Record 3P025-11-9v with the supervising engineer. The inspectors reviewed the test data for pump motor current, flow rate, differential pressure, and vibration in three planes at six locations. The data values were compared against the allowed values indicated on the Vision program trending plot and against the numerical limits on a hard

_ - _ _ _ _ _

- ._ -

.

-7-copy. The inspectors determined that measured performance placed the pump in the normal range for the licensee's inservice testing program. Therefore, the inspectors concurred with the licensee's determination that the pump was operabl Conclusions The postmaintenance and inservice testing performed by licensee personnel was effective and successfully validated that the Unit 3 swing component cooling water pump would perform its required safety functio M1.4 Excore Nuclear Instrument Surveillance Test - Unit 2 Insoection ScoDe (62707. 71707)

The inspectors reviewed the circumstances surrounding an excore nuclear instrument channel being retumed to an operable status without first completing the intended surveillance requirement Observations and Findings On December 7,1998, l&C technicians performed portions of a loop calibration procedure on Excore Channel C in Unit 2 to satisfy a biannual surveillance requirement in accordance with Procedure SO23-II-5.3, " Surveillance Requirement Excore Neutron Monitor Safety Channel C Channel Calibration," Revision 9. At the end of the day shift, the l&C technicians informed operators that they had finished for the day. Operators were aware that the work authorization allowed the l&C technicians to back out of the procedure and return it to an operable status. Because of a miscommunication, operators thought that the channel could be returned to service. Consequently, operators performed a channel check and declared Excore Channel C operabl On December 8, the l&C technicians returned to complete the biannual surveillance test and were surprised to find that the channel was not bypassed. After discussing the situation with the operators, l&C completeu a quarterly channel calibration surveillance in accordance with Procedure SO23-II-5.7," Surveillance Requirement N.I. Safety Channel C Drawer Test Linear Power Subchannel Gains Channel Functional Test and Channel Calibration," Revision 13, that confirmed the channel had remained operable in the as-found condition. The quarterly surveillance test was normally performed by l&C technicians when backing out of the longer biannual surveillance test prior to returning the channel to Operations as operabl The licensee initiated AR 981200460 to conduct an investigation into the circumstances and to identify appropriate corrective actions. The licensee determined that the channel ,

check was an adequate retest for the activities actually performed by l&C as part of the l biannual surveillance before operators declared the channel operable. However, a channel check was not considered to be an adequate basis for operability at all times when the biannual surveillance test was interrupted.

l

._

l

.

8-The inspectors discussed the specific circumstances of the test with Station Technica The licensee had determined that components inside the excore channel drawer had not been adjusted or touched and that only field portions of the loop had been monitore No external signal generators had been connected. Operators had confirmed, during the channel check, that the control room trouble alarm was not annunciated, which would have indicated that switch positions on the channel drawer were improperly positioned. The incpectors determined that, in this case, the channel check was an adequate retest, although the operators had not known that at the time the channel was declared operabl Conclusions A miscommunication between l&C technicians and operators resulted in an excore nuclear instrument channel being returned to an operable status without the intended retests being completed. For the work actually performed, the retest that was conducted was adequate to determine operability; however, the retest was not adequate for all the work that was authorized. Operators did not know that the channel had not been intended to be returned to service because the work was incomplet M2 Maintenance and Material Condition of Facilities and Equipment M2.1 Material Deficiencies - Unit 2 Inspection Scope (62707)

The inspectors walked down the Unit 2 EDGs and emergency core cooling system pump Observations and Findinas On November 25,1998, the inspectors observed that, on a Train A EDG 2G002 engine, a %-inch instrument air line had been rubbing against the %-inch air line to the air boost .

servo motor and severely eroded the %-inch line. The inspectors informed shift j supervision of the degraded line. The operators initiated AR 981101567 to document l and evaluate the deficiency. The licensee installed rubber and clamps on the line to l reduce further wear until the line could be replaced. The licensee appropriately documented the interim repair and line deficiency in a nonconformance report. The cognizant engineer performed further walkdowns of the remaining EDGs and initiated 1 ARs 981200471, -74, -77, and -78, which documented similar erosion deficiencie The licensee had identified on April 18 that an identical line on a Train B EDG 2G003 engine had broken because of wear at a retaining clip. The licensee had checked the clip tightness on all of the other EDGs as a result of this previous failure. The inspectors concluded that the licensee had missed an opportunity to identify the line rubbing wear i when looking for loose retaining clip j l

. . - _ . - . - _ - . . . _

.

On December 18, the inspectors identified that, on Pump 2P015, Train A low pressure safety injection, six bolts in a row of six were missing and that on Pump 2P016, Train B .

l Iow pressure safety injection, three of six bolts were missing. The bolts attached the vertical air plenum chamber to the upper air box assembly for the motor in two rows with six bolts to a row. At least six bolts on each chamber had remained properly installed, which held the chamber to the lower air box assembly with component cooling water piping providing lateral support. AR 981201133 documented the civil engineering seismic evaluation of the condition, which concluded t lat the pumps remained operable in the as-found condition. The inspectors determined that the evaluation demonstrain operability of the low pressure safety injection pumps. The licensee walked down the low pressure safety injection and containment spray pumps and documented additional fastener issues in ARs 981201152 and -5 On December 29, the licensee inspected the Pump 2P015 air chamber plenum, found no remnants of botting material, and replaced the missing bolts. The licensee  !

conducted a maintenance order search of work performed on the low pressure safet/

injection pumps and identified work that included air cooler plenum installation in 199 Consecuently, the licensee concluded that the missing bolts were inadvertently overlooked during the plenum installation. The licensee planned to evaluate whether further actions / assignments were warrante Conclusions The licensee missed opportunities to identify equipment deficiencies, in one case, a wear condition on an EDG control air system occurred that was nearly identical to a previoucty identified condition. In another esse, several fasteners were discovered to have been missing from two low pressure safety injection pump motor coolers for 5 years. The inspectors agreed with the licensee's conclusions that neither condition rendered the affected equipment inoperable. The licensee was responsive and thorough in resolving issues associated with the deficiencies, which did not affect operabilit M8 Miscellaneous Maintenance issues (92700)

M8.1 (Closed) Licensee Event Report 50-362/1997-003-01: pressurizer safety valve setpoir.ts out of toleranc The inspectors previously reviewed this issue in NRC Inspection Report 50-361:361/98-06, which resulted in identification of a noncited violation. The additional information provided in this licensee event report supplement did not change the characterization of the issue or affect the inspectors' conclusion l M8.2 (Closed) Licensee Event Report 50-361/1998-018-00: emergency core cooling system )

inoperable because of inadequate surve;llance tes On September 9,1998, engineers completed an evaluation of the totalloop uncertainty )

for the charging system flow instrumentation and determined that the uncertainty was l l

l

___ ____

7_

.

. .

-10-t 20 gpm. Because nominal charging flow was 44 gpm and Technical Specification Surveillance Requirement 3.5.2.6 required operators to verify that each charging pump develops a flow of greater than 40 gpm, the licensee concluded that the Technical Specification surveillance requirement had not been met. On September 10, within the 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> allowed by Technical Specification Surveillance Requirement 3.0.3, the licensee completed the required testing of the charging pumps by using more accurate meastte c nd test equipment. The inspectors reviewed the vendor information and licensee emculations that demonstrated that the use of the local measuring and test equipmer.t resulted in sufficient accuracy for the tes The licensee initiated AR 980900571 to evaluate the event. Until a permanent solution could be found, the licensee planned to perform future testing using local measurement and test equipment and calculating the charging pump flow, rather than using the installed charging loop flow instrumentation. The inspectors considered the planned corrective actions acceptable. The licensee concluded that, even if the charging flow had actually reduced, it would have resulted in a negligible increase in plant risk and had no actual safety consequence. The failure to ensure that adequate test instrumentation was used was a violation of 10 CFR Part 50, Appendix B, Criterion XI, " Test Control."

This nonrepetitive, licensee identified and corrected violation is being treated as a noncited violation, consistent with Section Vll.B.1 of the NRC Enforcement Policy (50-361;362/98019-01).

M8.3 (Closed) Violation 50-362: 362/97008-01: failure to ensure welding-induced sensitization of stainless steel would be minimize This violation resulted because the licensee failed to ensure that welding-induced e sensitization was minimized during gas tungsten arc welding on two Unit 3 safety-related pressure measurement nozzles for reactor coolant piping. Specifically, the welding procedure specified that limits for heat input variables were not measured or monitore During this inspection, the inspectors observed that the Nuclear Oversight Division took immediate action to establish interim monitoring of parameters associated with in-process welding activities. Within 2 weeks the licensee changed Procedure SO123-V-7.20.1, "ASME General Welding Standard," Revision 2, to include interim guidance for the monitoring of in-process welding parameters by welding engineers and Nuclear Oversight personnel. Procedure SO123-V 7.20.1, Revision 3 j required the determination of the appropriate parameters (volts, amps, and travel speed)

that would result in an in-process heat input below the welding procedure specification limit. The responsibility to ensure that heat input limits were not violated was assigned to the responsible work organization. Periodic monitoring of in-process variables by Nuclear Oversight and the welding engineer was encouraged. This violation is close l l

-

_ _ _ - _ _ _ _ _ _ _

.

l-11-

111. Enoineerina i E2 Engineering Support of Facilities and Equipment  !

E Root Cause Evaluation - Units 2 and 3 Inspection Scope (37551)

The inspectors reviewed Independent Safety Engineering Group Root Cause Evaluation 98-005, " Weed Resistance Temperature Detector (RTD) Failures," dated ,

December 17,1998, and attended a briefing of the preliminary results of the root cause '

evaluation on December Observations and Findinas  :

Weed RTDs were installed in several locations in the reactor coolant system of both units. Four premature failures were recently ev.perienced in the Unit 2 co.d leg, prompting the licensee to perform a detailed root cause evaluatio The licensee performed a mechanical vibration analysis of the Weed RTD installed inside a licensee-fabricated thermowell. The analysis measured the natural frequencies l of the thermowell and the RTD under the existing loose-fit condition and under a modified tight fit configuration. The data confirmed that the vibration was significantly reduced with a tight fit. The licensee also determined that the RTD wires were loose inside the RTD casing. The licensee made x-ray images of the RTDs and identified 1 stress points and flaws resulting from sharp edges in the ceramic casing. Scanning electron microscopic photographs showed that the breaks in the RTD wires resulted '

from ft.ti0u l l

The licensee contacted the vendor and learned that the vendor had determined similar I findings in response to problems encourtsred by other facilities. The vendor had j

'

modified the design of recently manufactured RTDs to stabilize the intemal wires and to eliminate the sharp edges that had nicked the wires previously. However, the vendor had nct notified the licensee of the design changes or of the problems experienced by other users. The licensee was discussing 10 CFR Part 21 implications with the RTD vendo As a result of the root causo evaluation, the licensee initiated actions to replace all the original six Weed RTDs installed with the newly designed Weed RTDs during the j upcoming refueling outage Conclusions The Independent Safety Engineering Group performed a rigorous and thorough root cause evaluation of tho failure of several RTD's, which demonstrated effective use of advanced technology. The recommendations of the root cause evaluation were corroborated by the vendo .

! -12-E2.2 Steam Generator Eaacrate Assessment - Units 2 and 3 Insoection Scope (TAC MA0587. TAC MA0588)

This section provides the NRC staff's final evaluation of the steam generator eggerate support issue identified by the licensee during the 1997 Unit 3 refueling outage. This evaluation applies to both Units 2 and Observations and Findinos Backaround During the 1997 Unit 3 refueling outage, the licensee discovered that portions of the steam generator eggcrate supports had degraded, which ranged from minor wastage of the eggcrate material to severe localized thinning. The significant degradation observed during the Unit 31997 refueling outage was confined mainly to the periphery locations of the eggerate supports. The secondary sides of the steam generators in both units were inspected during their 1997 refueling outages and during their 1998 midcycle outages. Significant degradation was limited to the periphery locations of the Unit 3 eggerate supports. The licensee plugged and stabilized by inserting a steel cable inside some Unit 3 steam generator tubes as a precautionary measure because of the  ;

degradation observed in certain eggerate supports. After removing the tubes from service, support from the eggerate structures was no longer needed. In NRC Inspection Report 50-361; 362/97-12, the staff reported on the licensee's June 5,1997, evaluation used to justify returning Unit 3 to power operation. The staff identified no deficiencies in the licensee's analysis of the adequacy of the eggerate support The licensee also investigated the Unit 2 steam generator eggerate supports after the degradation issue was identified in Unit 3. As reported in a March 10,1998, letter, the licensee observed minor isolated instances of thinning in the periphery areas of the Unit 2 steam generator eggerate supports; however, overall, the thinning was considerably less than that observed in Unit 3. No tubes in the Unit 2 steam generators were removed from service, as a result of eggcrate thinnin The NRC reviewed the program established by the licensee to conduct the video examinations of the eggerate supports during the Unit 21998 midcycle outage and reported its findings in NRC Inspection Report 50-361; 362/98-01. The primary difference between the inspection programs for the two units was that a larger portion of the Unit 3 eggerate structures was inspected. The staff documented that the scope of the Unit 2 secondary side visual inspections was satisfactory. Further, the results supported the conclusion that no steam generator tubes needed to be removed from service because of insufficient support from any secondary side support structures, l which included the eggerate supports.

,

l The licensee had extensively researched the cause of the eggcrete degradation and has '

concluded that the degradation resulted from a form of flow accelerated corrosion. Flow accelerated corrosion removes the protective oxide layer and exposes the base material

-13-to the fluid environment, which allows further material removal through corrosion and/or erosion. The carbon steel eggerate material utilized in the steam generators could be l susceptible to flow accelerated corrosion in the presence of sufficiently high fluid 1 velocitie l The licensee concluded that the flow accelerated corrosion in Unit 3 that occurred prior to its 1997 refueling outage resulted primarily from increased fluid velocities caused by l the buildup of deposits on the steam generator tubes. The deposits significantly reduced the available flow area within the tube bundle, causing flow diversion to the periphery of the tube bundle. The flow diversion to the periphery was also affected by the increased steam quality of the fluid within the tube bundle. Also, the buildup of I deposits changed the heat transfer characteristics of the tubes, causing the steam j quality to increase in the central region of the steam generators. These changes !

increased the flow resistance in the central portions of the steam generator, forcing l more flow to the peripheral regions with resulting higher velocities. The resulting large velocity gradients at the periphery initiated vortices that further elevated local velocities that were capable of dislodging the protective oxide layer of the eggerate material and initiating erosive flow accelerated corrosio l The chemical cleaning of the Units 2 and 3 steam generators during the 1997 refueling j outages removed the deposit buildup and restored fluid flow to nominal conditions, which reduced the high fluid velocities that led to flow accelerated corrosion and ,

stabilized eggcrate support degradation. The licensee changed the chemistry control program for the secondary systems to reduce the feedwater iron transport. This was

'

expected to prevent the level of deposit buildup that had been observed in the steam generators before chemical cleaning was done in 199 To confirm that the chemical cleaning of the steam generators stopped the flow accelerated corrosion and to ensure that no continued cignificant degradation of the ,

eggerate support structures went undetected, the licensee had committed to conduct j periodic inspections of the secondary side of the steam generators in both units during l future outages. By letter dated July 15, the licensee provided the results of the March 1998 midcycle cutage inspections of the Unit 3 steam generator secondary sid The licensee concluded that flow accelcrated corrosion had ceased and that no significant flow induced vibratinn had occurred since the 1997 refueling outage. The 1 27 steam generator tubes removed from service during the 1998 midcycle outage were l based on improved inspection techniques and use of conservative plugging criteria l rathar than progression of adaitional corrosio The NRC staff concurred that the flow accelerated corrosion resulted from deposit buildup on the steam generator tubes and that removal of the deposits should restore the steam generator secondary fluid flow io within nominal design values. The licensee i will conduct periodic inspections of the secondary side of the steam generators to check the level of deposit buildup on the tubes and to evaluate degradation of the eggcrate The licensee used this information to verify that tube integrity remained within the assumptions used in tha analysis to demonstrate continued operability of the steam generator .

-14- l l Steam Generator Operability Evaluation The final evaluation of the impact of degraded oggcrate supports on steam generator operability was submitted to the staff on October 17,1997. The licensee conservatively modeled the extent of eggerate degradation found during the Unit 31997 refueling outage in this evaluation. The licensee confirmed the adequacy of this model during followup inspections of the steam generator secondary sides. The licensee evaluated the ability of the Unit 3 steam generator eggcrate supports to perform their design functions for all design basis accident conditions and for normal operation condition As stated in the previous section, the amount of eggerate support degradation observed in Unit 2 was considerably less than that observed in Unit 3. Therefore, the staff concluded that demonstrating the ability of the Unit 3 steam generators to withstand a design basis seismic event would demonstrate the adequacy of the Unit 2 steam generators as wel b.3 Desian Basis Accident Evaluation In response to a 10 CFR 2,206 petition that questioned the ability of the San Onofre steam generators to safely withstand a seismic event, the staff evaluated that portion of the analysis pertaining to the design basis accident scenarios and reported its findings in a Director's Decision dated June 11,1998. The staff concluded in this Decision that the degraded eggerate supports would be capable of performing their design functions during and following all design-basis accidents, concurrent with a safe-shutdown earthquake, b.4 Normal Operatina Condition Evaluation The staff reviewed the analysis that demonstrated the ability of the degraded eggcrate structures to perform their intended functions during normal operating condition During normal operation, the eggerates are relied upon to provide an integrated support structure for the steam generator tubes to limit the effects of flow induced vibratio Flow induced vibration of steam generator tubes occurred as a result of flows either inside of the tubes (reactor coolant pump flow) or outside of the tubes (feedwater flow).

The effects of reactor coolant pump excitation on flow induced vibration in the tubes can be generated because of an impeller imbalance at a frequency of approximately 20 Hz or because of a pressure pulse resulting from vane interaction at 95-100 Hz. The licensee had reported on these effects in a letter dated October 17,1997. The licensee concluded that the resulting load because of a tube in resonance with 20 Hz pump vibration was only 0.25 g and that the resulting stress in the tube at resonance .

frequency because of a pressure pulse at a frequency of 95-100 Hz was less than 2 ks !

The licensee considered both of these results to be negligible and the staff agrees with this assessmen Flow induced vibration of tubes because of feedwater flow on the outside of the tubes could result from vortex shedding, turbulence, and fluid-elastic instability. The licensee stated that the vortex shedding phenomenon is generally considered insignificant in two-phase flow regimes, such as the flow inside a steam generator. The licensee also

.. .. . - . -.

'

l l

I

! -15- i l

stated that the tube vibration resulting from turbulence is generally small. The staff agreed with the assessment that vortex shedding and turbulence will not result in significant loads on the tubes.

l

'

The licensee investigated the possibility of fluid-elastic instability affecting steam generator tube integrity under degraded eggerate suppcit conditions. Absence of j eggerate support would reduce the natural frequency of a steam generator tube, which increases the potential for tube instability in either a cross flow or a parallel flow regim The tube stability evaluation was based mainly on the current theory of flow-induced vibration, developed by Connors. Some of the parameters appearing in the l l mathematical formulas, e.g., the instability constant and critical damping ratio, were !

determined based on laboratory testin The steam generator tubes with the largest unsupported spans, resulting from eggerate j

'

degradation, were identified as the most critical tubes in the bundle for the flow induced !

vibration evaluation based on their low natural frequencies. Susceptibility of these critical tubes to fluid elastic instability was assessed by calculating the stability ratio,

,

defined as the ratio of the effective flow velocity (calculated based on the actual flow l

!

velocity) and the critical velocity (velocity when the tube would be susceptible to flow j induced vibration). A stability ratio of 1.0 corresponds to potential instability, and a lower ;

l value of the stability ratio corresponds to a higher margin of stability. Lack of eggerate

, support at one or more consecutive locations can lead to higher stability ratios for many tubes, depending on the flow regime and regio The licensee found that significant eggerate erosion appeared only on the hot side of the steam generator where the highest fluid velocities also exist. Therefore, only the hot

, side of most of the selected tube rows was modeled, which was sufficient to develop the

'

relevant mode shapes. However, the licensee used full models (hot and cold side) for I

'

the lower tubes because of the vertical tube support grid arrangement. The licensee also selected tube rows for evaluation based on geometry and flow regime. Nineteen tube rows were selected since they would envelope the other tubes in terms of frequency response and velocity. The staff found the tube selection process used by I the licensee acceptable, as it bounded the efiects of fluid-elastic instability on the remainder of the tubes.

I

! The results of the calculation of flow-induced stability for selected tube rows indicated that twelve tube rows had stability ratios above the limit of 1.0 when the top two eggerates were assumed to be ineffective. Tube rows 83 and 110 were the most critical as they exhibited a stability ratio of 1.13. When these two rows were evaluated for a l single ineffective eggerate, the stability ratios dropped to 0.59 and 0.64, respectively.

!

This showed that the remaining tube rows will exhibit even greater margins against instability with only one inactive eggerate support. In many locations with more than one eggerate not credited, a stability ratio less than 1.0 was calculated.

f in order to prevent the possibility of unacceptable fluid-elastic instability conditions within the tube bundle, the licensee established criteria for removing all tubes from service 4 (i.e., the tubes would be plugged and stabilized). This criteria specified removing tubes

!

. .

.

.

-16-from service, for which more than one consecutive ineffective eggerate support was found. The staff considered this approach acceptable since the stability ratios for the tubes that remain in service was less than The staff concluded that this approach used to assess steam generator tube flow induced vibration under degraded eggcrate support conditions is acceptable. Also, the licensee determined that no significant flow induced vibration had occurred since the 1997 refueling outage. This provided additional assurance that the steam generator internal supports, with the degraded eggcrate support structures, were capable of performing their design functions during normal operating condition Conclusions The NRC staff reviewed tne inspection techniques and results, the analytical assumptions and methodology, and the acceptance criteria used by the licensee to evaluate the impact of the degraded eggerate supports discovered in the San Onofre Unit 3 steam generators. The staff concluded that the amount of degradation observed in the eggcrate supports will not prevent the Units 2 and 3 steam generators from i performing their intended safety function E4 Engineering Staff Knowledge and Performance E4.1 Core Operatina Limits Suoervisorv System (COLSS) Addressable Constant Error - j Unit 2 1 Inspection Scoce (37551)

!

The inspectors reviewed the circumstances surrounding the licensee's l l

November 21,1998, discovery that a COLSS addressable constant contained an incorrect value. The inspectors reviewed AR 981101362, including the operability ,

assessment, and discussed the issue with the computer engineering superviso l Observations and Findinos The licensee found that COLSS constant K9957, the Part Length Control Element Assembly (PLCEA) Unpenalized Planar Radial Peaking Factor, had a value of 0.1556 instead of 1.566. The licensee determined that the condition had existed since i November 18,1998, when a software modification had been implemented. The l engineer who implemented the software modification was aware that several constants j

'

had to be reentered as a result of the modification and had reentered them all. The engineer did not recognize the entry error during self-verification of the entrie A computer technician had performed an operability surveillance of the COLSS constants in accordance with Procedure SO23-3-3.41,"COLSS/COLSS Backup Computer System Operability and Computer Surveillances," Revision 3. However, the computer technician also missed the entry error. Step 2.7.1.7 of Attachment 1 required I

.

-17-that the technician " verify the latest Log Book value for each point is the same as the value on the printout." The technician incorrectly performed this step and marked it as satisfactorily complete The inspectors reviewed the controls for addressable constants and reviewed the historical recorf A incorrect addressable constants. The inspectors determined that, historically, the controls for COLSS constants had been effective. The licensee determined that the current errors were individual performance errors and counseled those involved. The licensee also identified some procedural enhancements to help reduce such errors. Most significant was the addition of a requirement to use a printout of the constants, which is formatted the same as the log of approved constants, when performing the verification after making database changes. The failure to follow procedures for entry and val:dation of COLSS constants was identified as a violation of Technical Specification 5.5.1.1.a. This nonrepetitive, licensee-identified and corrected violation is being treated as a noncited violation, consistent with Section Vll.B.1 of the NRC Enforcement Policy (50-361; 362/98019-02).

The operability assessment determined that the error affected the COLSS power limit calculation while PLCEAs were inserted into the core. The PLCEAs were inserted into the core for several hours during the period of concern in support of a reactor power reduction to heat treat the circulating water system. The constant was corrected before reactor power was returned to 100 percent, and the PLCEAs were withdrawn, which provided excellent "before and after" data for assessment purposes. The data clearly showed that the COLSS-calculated power limits for departure from nucleate boiling and, to a lesser extent, for linear heat ratio, were nonconservatively affected by the incorrect constant. However, the licensed power limit, which was unaffected, would have been ;

reached before the calculated power limits. Since none of the power limits were i exceeded, no actual safety consequence resulte The potential consequence of this specific error was small, because of the actual value inserted for the constant and because of the margin between the licensed power limit I and the calculated power Smits. However, the licensee did not assess the potential '

consequence of this type of error. The COLSS is not safety-related although the COLSS calculations are used to allow decreased operating margins to the thermal limits; hence, errors in the calculations could result in operating with less margin than allowed. The procedures for control and surveillance of the COLSS constants were classified as quality affecting and were subject to the requirements of 10 CFR Part 50, l Appenoix B. Therefore, the inspectors determined that the potential consequence of this type of error was significant.

I c. Conclusions A noncited violation of licensee procedures and Technical Specification 5.5.1.1.a was identified, consistent with Section Vll.B.1 of the Enforcement Policy. Specifically, a technician failed to properly verify that one COLSS constant was consistent with the log of approved constants.

l l

. ._ _ .

.

-18-E7 Quality Assurance in Engineering Activities E7.1 Stoo Work Order for Fuel Fabrication Insoection Scope (37551)

The inspectors reviewed the circumstances surrounding the licensee's rescindment of the July 7,1998, stop work order to the fuel vendor for the fuel manufacturing proces Observations and Findinas During inspection of fuel assembly zircaloy grids for the Unit 2, Cycle 10 fuel load, the nuclear fuel vendor identified differences in the appearance of some of the seam and lap welds on the grids. Ten grids are installed in each fuel assembly to provide the lateral structural support for the fuel pins in the fuel assemblies. Each grid included dozens of welds, ar.d only a few welds on some grids exhibited the different appearance. The differences were later determined to have resulted from stainless steel inclusion in ;he zircaloy weld material. The apparent cause was improperly focused lasers drawing material from the stainless steel weld fixture into the welds. Fuel vendor personnel examined the grids and classified the weld contamination as light, moderate, or heavy, based on subjective criteria. Subsequently, objective criteria for tha classifications were develope When the fuel vendor informed the licensee of the contamination problem, the licensee dispatched personnel to the fuel manufacturing facility to independently inspect the grids and assess the condition. The licensee identified that a large percentage of the grid j classifications were less conservative than classifications based on earlier objective criteria. Therefore, the licensee determined that the fuel vendor had not confirmed that the previously completed classifications were consistent with the objective criteria. Also, 20 fuel assembly cages, using 10 potentially contaminated grids each, had been  ;

shipped to a separate fuel vendor fabrication facility to be assembled into fuel l assemblies. Some of the grids could not be inspected because they had already been I installed in fuel assemblies. On July 7, based on these discrepancies, the licensee  !

'

issued a stop work order to the fuel vendor pending resolution of the quality issues. The vendor continued production of components and fuel assemblies, at risk of licensee rejection, while developing and implementing a dispositioning pla A letter dated October 21 formally withdrew the stop work order. The attachment to this letter indicated that the vendor had: (1) developed screening criteria to identify defective welds, (2) developed inspection criteria to assure design requirements were met, (3) performed testing of grids for material strength and corrosion resistance, and (4) engaged a third-party to perform an independent assessment. The inspectors reviewed documentation provided by the licensee and held discussions with quality assurance and fuel performance analysis personnel to evaluate the validity of the

. licensee's rescindment of the stop work orde ~. _ _ . _ . _ . . _ _ ._._ _ . _ . _ _ _ _ _ _ _ . . . _ . . _ _ _ _ _

=

!

l

.

-19-The fuel vendor performed mechanical strength and chemical testing on contaminated grid specimens. The specimens were selected to bound the entire population of Unit 2 Batch M fuel. The testing showed that grid welds with higher levels of contamination met all functional and design requirements. Additionally, chemical testing and analysis revealed that weld contamination was limited to stainless steel constituents and would i not impact the fortgevity of the weld )

According to licensee representatives, the fuel vendor had taken extensive corrective action to implement and maintain correct laser focus, improved welding procese, and post-weld inspection. The representatives also stated that the vendor's corrective action, and the results of the test and inspection effort, provided the basis for rescinding the stop work order. In addition, no further inspection, beyond the normal fuel receipt inspection at the facility would be require '

' Conclusions The Nuclear Oversight and Nuclear Fuels groups had effectively interacted with the nuclear fuel vendor to assure that fuel scheduled for loading into the reactor cores met the design and functional requirements. The action taken to rescind the stop work order that was ;ssued to the vendor on July 7,1998, was appropriat IV. Plant Support j R1 Radiological Protection and Chemistry Controls R Housekeepina Inspection Scope (71750)

The inspectors walked down accessible portions of Units 2 and 3 radiologically controlled areas to assess the status of housekeepin Observations and Findinas No housekeeping deficiencies were identified during the walkdowns, except an unsecured ladder in the Unit 3 tendon gallery (Section O2.1). Conclusions Housekeeping in the radiologically controlled area was consistent with licensee procedure I t

.

.

-20-V. Manacement Meetinas X1 Exit Meeting Summary The inspectors presented the inspection results to members of licensee management at the exit meeting on January 8,1999. The licensee acknowledged the findings presente The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identifie I V, h ,

!

.

- -

-_ _ _ - - _ __ _ _

.

-

. ATTACHMENT SUPPLEMENTAL INFORMATION L

PARTIAL l.lST OF PERSONS CONTACTED Licensee :

D. Brieg, Manager, Station Technical J. Fee, Manager, Maintenance G. Gibson, Manager, Compliance D. Herbst, Manager, Site Quality Assurance J. Hirsch, Manager, Chemistry R. Krieger, Vice President, Nuclear Generation

- J. Larson, Supervisor, Procurement Quality J. Madigan, Manager, Health Physics

_

D. Nunn, Vice President, Engineering and Technical Services

^ T. Vogt, Operations Plant Superintendent, Units 2 and 3

,

R. Waldo, Manager, Operations INSPECTION PROCEDURES USED ,

IP 37551: Onsite Engineering

. IP 61726: Surveillance _ Observations j IP 62707: Maintenance Observations .l'

. IP 71707: Flant Operations IP 71750: Plant Support Activities IP 92700: On Site LER Review IP 92901: Followup - Operations TAC MA0587 Steam Generator Eggerate Assessmen TAC MA0588 - . Steam Generator Eggerate Assessment ,

j ITEMS OPENED AND CLOSED Ooened and Closed 361; 362/98019-01 NCV charging pump Technical Specification surveillance requirement not met (Section M8.2)

361; 362/98019-02 NCV COLSS surveillance procedure not properly followed (Section E4.1)

Closed 361; 362/97020-01 VIO failure to provide special corrective lenses for use in {

self contained breathing apparatus (Section 08.1)

l

!

.

. . .

_ _ _ _ _ _

-

.

.

.

-2-362/1997-003-01 LER pressurizer safety valve setpoints out of tolerance (Section M8.1)

361/1998-018-00' LER emergency core cooling system inoperable due to inadequate surveillance test (Section M8.2)

362; 362/97008-01 VIO -failure to assure welding-induced sensitization of stainless steel

! would be minimized (Section M8.3)

LIST OF ACRONYMS USED AR action request CFR Code of Federal Regulations COLSS core operating limit supervisory syster:

EDG emergency diesel generator l&C instrumentation and Control NCV noncited violation NRC Nuclear Regulatory Commission PLCEA part length control element assembly RTD resistance temperature detector VIO violation

!

$

9,

' ' ' '

. _ _ _ _ . _ _ _ _ _ _ _