IR 05000528/1998015
| ML17313A566 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 09/02/1998 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17313A565 | List: |
| References | |
| 50-528-98-15, 50-529-98-15, 50-530-98-15, NUDOCS 9809100033 | |
| Download: ML17313A566 (36) | |
Text
ENCLOSURE U.S. NUCLEAR REGULATORYCOMMISSION
REGION IV
Docket Nos.:
License Nos.:
Report No.:
Licensee:
Facility:
Location:
Dates:
Inspectors:
Approved By:
50-528; 50-529; 50-530 NPF-41; NPF-51; NPF-74 50-528/98-15; 50-529/98-15; 50-530/98-15 Arizona Public Service Company Palo Verde Nuclear Generating Station, Units 1, 2, and 3 5951 S. Wintersburg Road Tonopah, Arizona May 21 through August 6, 1998 William P. Ang, Senior Reactor Inspector, Maintenance Branch Rebecca L. Nease, Senior Reactor Inspector, Engineering Branch Dr. Dale A. Powers, Chief, Maintenance Branch Division of Reactor Safety ATTACHMENT:
Supplemental Information 9809100033 980902 PDR ADQCK 05000528
-2-EXECUTIVE SUMMARY Palo Verde Nuclear Generating Station,,Units 1, 2, and 3 NRC Inspection Report 50-528/98-15; 50-529/98-15; 50-530/98-15 On May 20, 1998, a Palo Verde Nuclear Generating Station, Unit 1, 2-inch charging and volume control system letdown pipe cracked and leaked approximately 175 gallons of primary coolant into the auxiliary building valve gallery. An additional 325 gallons of letdown flowwas diverted to the equipment drain tank via a lifting relief valve, PSV-345.
NRC Region IV dispatched two inspectors to Palo Verde Nuclear Generating Station to perform an inspection of the event. The objective of the inspection was:
(1) provide prompt assessment and inspection of the Unit 1 charging and volume control system letdown line crack and leak, (2) assess licensee immediate and root cause evaluations associated with the event, and (3) assess licensee corrective actions for the event.
The inspectors used NRC Inspection Procedure 93702 to perform the inspection and subsequent followup inspections.
Overall, the inspectors determined that licensee personnel responded appropriately to the event and took appropriate prompt corrective actions.
~0erations
~
Control room personnel responded to the event appropriately.
Appropriate evaluations associated with the immediate consequences of. the event were performed by licensee engineers (Section 01.2).
~
Operations personnel and engineers performed appropriate. evaluations of the conditions experienced during the Unit 1 letdown pressure oscillation event and initiated appropriate procedural guidance to minimize the occurrence of similar events (Section 01.3).
Maintenance
~
Maintenance personnel appropriately replaced the damaged letdown piping and pipe supports.
A lack of an adequate valve lineup initiallyprevented venting of the piping that resulted in a defect in the pipe weld and required replacing the pipe a second time (Section M1.2).
Enrntneerinq Engineers performed appropriate evaluations of the failed letdown line to bound the conditions during the event and to allow an interim repair of the piping (Section E1.1).
The failed letdown line and Pipe Supports 13-CH-037-H-OOA and 13-CH-037-H-OOB were appropriately redesigned (Section E1.2).
'I
-3-The nonrepetitive, licensee-identified, low-safety significant, failure to install the Updated Final Safety Analysis Report, Chapter 15.6.2, described letdown flow hi/lo alarm in all three Palo Verde units was identified as a noncited violation (EA 98-383) pursuant to Section VILB.1 of the NRC Enforcement Policy (Section E1.3).
r Licensee engineers were performing comprehensive, albeit lengthy, evaluations of numerous facets of the event to determine the equipment root cause of failure. The completed equipment root cause report of failure analysis was not available during the inspection (Section E1.3).
Re oit Details Summa of Plant Status All three units were operating at approximately full power during the onsite inspection period.
Conduct of Operations 01.1 Char in and Volume Control S stem Letdown Line Crack Se uence of Events On May 20, 1998, Unit.1 was in Mode 1 at approximately 100 percent power.
At approximately 12:59 a.m., the quarterly charging pump operability surveillance test for Charging Pump E was completed in accordance with Surveillance Procedure 40ST-9CH06.
Charging Pump A was started to restore the unit to its normal two-pump operation.
Accordingly, letdown flowwas increased by manual manipulation of letdown Flow Control RCN-LIC-110 to throttle open letdown flow Control Valve 1CH-LV110Q. Letdown heat exchanger backpressure Control Valve 1CH-201P was in service and in the automatic control mode.
Subsequently, control room personnel observed indications of letdown flow mismatch.
Control room personnel also observed indications that letdown Relief Valve PSV-345 may have lifted and letdown flow and pressure perturbations were observed.
At 1:06 a.m., auxiliary building Radiation Monitor RU-9 alert alarm was received in the control room. Operators placed the backpressure Control Valve 1CH-201P Controller CHN-PIC-201 in manual and increased the valve demand to 50 percent open.
The operators observed that this action dampened the observed pressure oscillations.
Controller CHN-PIC-201 was subsequently returned to the automatic mode.
Approximately 2 minutes later, the pressure oscillations recurred.
At 1:09 a.m., a second auxiliary building Radiation Monitor RU-8 alert alarm was received in the control room: At 1:10 a.m., a Radiation Monitor RU-9 high alarm was annunciated.
Operations personnel believed that a valve packing leak had occurred in the auxiliary building and initiated the actions required by the alarm response procedure.
Radiation protection personnel were contacted by control room operators to confirm the validity of the alarm. An auxiliary operator and a radiation protection monitor were dispatched to the auxiliary building to attempt to identify the source of the lea The auxiliary operator notified the control room that steam and water were observed coming from the letdown valve gallery area.
At 1:23 a.m., control room operators shut.
letdown Isolation Valve CHV-UV-515 and isolated letdown. Control room personnel initiated actions required by Abnormal Operating Procedure 40AO-9ZZ05, "Loss of Letdown." The Unit remained in Mode 1st approximately 100 percent power.
Prom t Onsite Res onse Ins ection Sco e 93702 NRC Region IVdispatched two inspectors to the site to determine the facilitystatus relative to the event and to determine the need for further recommended immediate actions.
The inspectors were briefed by the licensee's incident investigation team director, and other licensee personnel, regarding the event, plant status, and ongoing activities associated with the event. The inspectors reviewed and discussed Condition Report 1-8-0322 that was initiated for the event with the director and other licensee personnel.
The inspectors reviewed plant computer data and control room logs associated with the event.
Observations and Findin s The inspectors determined, upon arrival onsite on May 21, 1998, that Unit 1 continued to be in Mode 1 and was operating at approximately 100 percent power. The inspectors confirmed that the charging and volume control system letdown line leak had been stopped and the letdown line had been isolated.
The inspectors confirmed that control room operators were performing the actions required by Abnormal Operating Procedure 40AO-9ZZ05, "Loss of Letdown." The inspectors confirmed that control room operators were monitoring reactor coolant pump and seal parameters due to the loss of charging and reactor coolant pump seal injection that resulted from the isolation of letdown. Reactor coolant pump seal temperatures and pressures increased during and after the event while letdown was isolated.
Engineering personnel performed an evaluation of the reactor coolant pump seals, as part of the incident investigation, and
, determined the temperatures and pressures experienced by the seals to be within acceptable ranges.
The inspectors confirmed that control room operators complied with applicable Technical Specifications associated with the event.
Control room logs indicated that Action required by Technical Specification Limiting Condition for Operation 3.0.3 was entered at 1:06 a.m. on May 20, 1998, due to a pipe break on the letdown line which was not isolated as required by Action b of Limiting Condition for Operation 3.4.9,
"Reactor Coolant System Structural. Integrity." The technical specification action statement was exited at 1:23 a.m.'when letdown was isolate jl
-6-The inspectors confirmed that operations personnel performed immediate reportability reviews as part of the condition report initiation. Control room personnel determined that the event was not immediately reportable in accordance with 10 CFR 50.72.
Nonetheless, Nuclear Regulatory Affairs personnel notified the NRC resident inspectors of the event.
The inspectors reviewed two safety assessments performed by the transient a'nalysis group of the licensee's nuclear fuel management department and discussed the assessments with licensee engineers.
The assessments were documented in Memoranda 162-08369-KCP/AMT, "NFM Nuclear Safety Assessment:
Unit 1 Letdown Break on May 21, 1998," issued May 21, 1998 (erroneously dated March 21, 1998) and 162-08370-KCP/HAT, "Safety Assessment of Unit 1 Letdown Line Isolation,"
dated May 21, 1998. The first assessment reviewed the impact of the event on nuclear safety and ensured that the consequences of the event were bounded by analysis contained in the Palo Verde Nuclear Generating Station Updated Final Safety Analysis Report.
Updated Final Safety Analysis Report, Chapter 15.6.2, documented an analysis of a double-ended break of the letdown line outside the containment.
The safety assessment concluded that the event did not result in a transient that was more severe than what was already analyzed and documented in the Updated Final Safety Analysis Report. The safety assessment also concluded that the dose consequences due to the event remained well below the 2-hour exclusion area boundary thyroid dose consequence of 22.4 Rem for the Updated Final Safety Analysis Report, Chapter 15.6.2, analyzed double ended letdown line break. The second assessment determined the safety significance of operating Unit 1 with the letdown line isolated.
Various Updated Final Safety, Analysis Report, Chapter 6 and 15, analyzed events were considered including a feedwater line break, loss of condenser vacuum, inadvertent deboration and various loss-of-coolant events.
The assessment concluded that Updated Final Safety Analysis Report safety analysis results would not be adversely impacted by operating Unit 1 'with the letdown line isolated provided no more than two charging pumps were simultaneously in service.
The inspectors determined that reasonable assessments were performed by the transient analysis group.
The inspectors were informed by the incident investigation director, who also was the Unit 1. operations department leader, that the director had issued a night order to all operations department personnel to inform them of the Unit 1 letdown line event. The inspectors reviewed the Units 1, 2, and 3, night order dated May 20, 1998, and confirmed that a description of the Unit 1 letdown line event was provided to the operations department personnel.
The inspectors also noted that the night order cautioned operating crews about the uncertainties associated'with the controls of the letdown backpressure control valves. The night order required operating crews to obtain the operations department leader authorization for any planned activities that may challenge the letdown backpressure control valve Conclusions The inspectors concluded that control room personnel responded to the event appropriately.
The inspectors also concluded that appropriate evaluations associated with the immediate consequences of the event were performed by licensee engineers.
Letdown Restoration Procedure lns ection Sco e 93702 The inspectors reviewed and discussed with operations personnel and licensee engineers the procedures used for restoration of letdown.
Observations and Findin s The inspectors determined that, the operability test for Charging Pump E was performed using Procedure 40ST-9CH06, "Charging Pump Operability Test," Revision 3. At the conclusion of the test, Section 9.0, "System Restoration," of the procedure required the charging pumps to be restored to the operational configuration desired by the control room supervisor/shift supervisor.
Charging Pump A was selected and started using Procedure 41OP-1CH01, "Charging and Volume Control System Normal Operations,"
Revision 28. Section 11.3, "Changing the Running Order of Charging Pumps," of the procedure provided the details for starting a second charging pump. The procedure did not contain any precautions for letdown backpressure osciliations. When letdown was isolated during the event, operators initiated the procedural requirements of Abnormal Operating Procedure 40AO-9ZZ05, "Loss of Letdown," Revision 3. The procedure provided the instructions for control of pressurizer level while letdown was isolated, and the instructions for restoration of letdown flow.
After the damaged letdown pipe and pipe supports were replaced, letdown was restored using Appendix A, "Restoration of Letdown with a Pressurizer Steam Bubble." Licensee engineers and operations personnel refined the procedural steps by preparing a letdown restoration gameplan document, dated May 22, 1998, to provide control room operators further guidance for accomplishing the abnormal operating procedural steps.
The document provided guidance to start the second charging pump with 200 to 450 psig letdown pressure and with the backpressure control valve in manual.
The inspector observed the restoration of.letdown performed by Unit 1 operators on May 22, 1998. The inspector observed good interaction between operators and engineers prior to and during the restoration.
The inspector observed a good briefing of operators that was conducted prior to the restoration.
The inspectors noted significant operations department management involvement during the restoration.
Operators initiallyplaced Charging Pump A in service.
Backpressure Control Valve CHN-V201P was in manual control. Letdown backpressure was stabilized at approximately 200 psig before Charging Pump B was started and letdown flowwas slowly increased.
The inspector noted that letdown backpressure spiked approximately 210 psig to a letdown backpressure of approximately 410 psig when Charging Pump B was started and letdown flowwas increased.
Operators manuaHy operated the backpressure Control
-8-Valve CHN-V201P and stabilized backpressure at 200 psig before slowly increasing the backpressure to the desired 440 psig normal backpressure.
The inspector noted that the letdown relief valve did not open because the starting back pressure of 200 psig, combined with the 210 psig pressure spike that occurred when letdown flowwas increased, did not reach the 600 psig setpoint of the relief valve. The inspector noted that subsequent letdown backpressure perturbations were minimal because of the manual control of the letdown back pressure control valve and the absence of pressure fluctuations that normally accompany relief valve opening and closing.
The inspectors discussed with licensee operations personnel and engineers the letdown pressure oscillations that occurred during the event and the pressure spike observed during the Unit 1 restoration of letdown. Licensee operations personnel acknowledged the need for additional letdown pressure control during evolutions that involve increasing the number of running charging pumps and increasing letdown flow. Operations personnel informed the inspectors that a review of procedures had been performed and the following procedures would be changed to add the needed letdown pressure controls.
40ST-9CH06, "Charging Pump Operability Test," Revision 3 4XOP-1CH01, "Charging and Volume Control System Normal Operations,"
Revision 28
. 4XST-1 CH04,;Boron Injection Flow Test," Revision 10 40OP-9CH13, "Charging Pump Pulsation Dampener Operation," Revision 3 c.
Conclusions The inspectors concluded that operations personnel and engineers performed appropriate evaluations of the conditions experienced during the Unit 1 letdown pressure oscillation event. The inspectors concluded that appropriate procedural changes were being'nitiated.
II. Maintenance NI1 Conduct of Maintenance M1.1 Modification of Unit 1 Letdown Pi in and Pi e Su orts a.
Ins ection Sco e 93702 The inspectors reviewed and discussed with licensee engineers and maintenance personnel the modification of the Unit 1 letdown piping and pipe support b.
Observations and Findin s Maintenance personnel removed approximately 18 inches of the existing damaged letdown piping and pipe supports and installed the new pipe piece and pipe supports in accordance with Work Order 00841404.
During welding of one end of the replacement pipe, a pinhole leak, accompanied by escaping gas, occurred.
The escaping gas ignited, creating a plume approximately 8 inches long. The flame was immediately extinguished by covering the pinhole. Condition Report/Disposition Request 1-8-0335 was written. Subsequent review by licensee personnel determined the gas to be hydrogen.
Licensee personnel also determined that the vent path for a required purge prior to welding was blocked by normally-open Valve 1PCHNV342, which was found to be shut.
Review of a clearance tag assignment sheet for the work order showed that the valve was not listed for verification or reposition.
The licensee identified lack of a procedural requirement for establishing an appropriate vent path was of low nuclear safety significance.
Maintenance personnel cut out the new replacement pipe piece and replaced the pipe in accordance with Work Order 00841404.
The NRC inspectors reviewed documentation of the completed Work Order 00841404.
The inspectors determined that damaged letdown piping and pipe supports were replaced in accordance with the design requirements.
Conclusions Maintenance personnel appropriately replaced the damaged letdown piping and pipe supports.
A lack of an adequate valve lineup initiallyprevented venting of the piping that resulted in a defect in the pipe weld and required replacing the pipe a second time.
III. En ineerln E1 Conduct of Engineering E1.1 En ineerin Walkdownof Letdown Pi in a., lns ection Sco e 93702 The inspectors reviewed and discussed with licensee engineers the design and failed condition of the Unit 1 letdown line that cracked and leaked.
The inspectors performed a visual inspection of the damaged pipe, pipe support, and the affected piping system.
b.
Observations and Findin s As part of the licensee's incident investigation, the licensee engineers performed an evaluation of the letdown line failure and issued an engineering evaluation report on June 26, 1998. The evaluation documented the engineering inspections and preliminary
I II l
-10-assessment of the equipment root cause of failure, the type and scope of repair performed, justification for commencement of letdown operation in Unit 1 following repair, and transportability review for Units 2 and 3. The inspectors reviewed and discussed with licensee engineers the evaluation report.
At approximately 10:00 a.m. on May 20, 1998, an engineering team performed an inspection of the affected portions of the letdown line. Five feet of insulation was removed, prior to the inspection, between Pipe Support 1CH037HOOA and the penetrations to the letdown heat exchanger room and the upstairs valve gallery, where Valve CH-345 was located.
The inspection noted the following:
~
Only slight damage to insulation was observed over the majority of the remaining letdown line insulation. However, in the penetration to the letdown heat exchanger room, insulation was observed to be damaged.
~
The'letdown line cracked at the interface between the line and the trunnion of Pipe Support 1CH037HOOA that was welded to the line in accordance with its design.
~
Significant damage to the pipe support was observed.
The trunnion was bent, the spring can had failed, the vertical position of the trunnion, in relation to the spring can, no longer supported the weight of the pipe.
Pipe Support 1CH037HOOB, which consisted of a "strap" around the damaged trunnion, had the paint scraped off its top surface.
Additional insulation was subsequently removed and the remainder of the affected portion of the letdown line was inspected..
Licensee engineers observed the following:
A 3-foot section of the letdown line, centered on Pipe Support 1CH037HOOA was bowed in an upwards direction.
No rejectable defects were observed by visual examination of letdown line buttwelds.
~
Avertical section of the letdown line downstream of the crack appeared to have been in contact with a piece of nearby structural steel as evidenced by paint that appeared to have been rubbed off the structural steel.
Based on the results of the inspection, licensee engineers concluded that the piping system experienced a dynamic transient of sufficient magnitude to cause observable pipe support deformation.
Licensee engineers noted that complex hydraulic forces were involved in the dynamic event. The magnitude of the loads that had to have occurred was determined by superimposing dynamic loads on the existing stress model for the piping. The loading was increased until piping material yield strength values were exceeded.
The analysis determined that a superimposed 1000-pound force applied in the axial direction and a 500-pound force applied in the lateral direction at the elbow immediately upstream of the damaged location, produced pipe stresses and support
f
-11-loads sufficient to cause the observed deformation. The pipe stress and support load was subsequently used as an additional load design configuration for modifying and redesigning the piping and pipe supports for Unit 1. The damaged pipe was sent to an independent failure analysis company for detailed examination and failure analysis.
The inspectors reviewed the following:
Drawing 01-P-CHF-201, "AuxiliaryBuilding Isometric, Chemical and Volume Control System, Letdown Line to Letdown Heat Exchanger," Revision 3 Drawing 01-M-CHP-001, "P 8 I Diagram, Chemical and Volume Control System,"
Revision 17 Drawing 13-CH-037-H-OOA, "Pipe Support Assembly," Revision 2 Drawing 13-CH-037-H-OOB, "Pipe Support Assembly," Revision
Stress Calculation 01-MC-CH-507, "Chemical and Volume Control System, Letdown Line to Letdown Heat Exchanger," Revision 0 The inspectors noted that the failed pipe was a horizontal portion of Line N-037-FCBA.
The failed letdown line pipe was an ASME,Section III, Class 2, 2-inch Schedule 40S, Type 304 pipe. The letdown line pipe was designed for 650 psig and 550'F. The inspectors noted that the letdown line pipe failed at the connection to tandem Pipe Supports 13-CH-037-H-OOA and 13-CH-037-H-00B.
Pipe Support 13-CH-037-H-OOA was a variable spring support that vertically supported the pipe through a base plate and a 1-inch Schedule 40 stainless steel pipe that was welded to the bottom of the letdown line. Pipe Support 13-CH-037-H-OOB was an axial restraint that restrained the letdown line by restraining Pipe Support 13-CH-037-H-OOA by means of "U" shaped structural member that was wrapped around the 1-inch vertical pipe, 3-3/1 6 inch below the centerline of the letdown line. The inspectors noted that typical axial restraints restrained piping by means of lugs welded to the pipe that minimized the moment arm of the restraint. The inspectors noted that the Pipe Support 13-CH-037-H-OOB restraint design incorporated a moment arm that was subject to construction tolerances and consequently multiplied the forces experienced at the letdown line connection.
The inspectors determined that the pipe support design met ASME code requirements.
. However, the inspectors considered the Pipe Support 13-CH-037-H-OOB design to be poor due to the incorporation of a moment arm that exacerbated restraining forces on
. weld attachment to the letdown line.
The inspector performed a visual inspection of the Unit 1 letdown piping. At the time of the inspection, the failed portion of the letdown line and the attached pipe support stanchion had already been cut out and stored in the Unit 1 hot machine shop. The inspector visually inspected the failed portion of the letdown line at the hot machine shop. The inspector confirmed the licensee engineer's walkdown inspection results.
The inspector noted that the letdown line had intermittent cracks around the circumference of the toe of the attachment weld between the 2-inch letdown line and the 1-inch pipe support trunnio t
-12-Licensee engineers performed a visual and liquid penetrant inspection of the Units 2 and 3 trunnion-to-pipe welds of Hangers 2/3CH-037-H-OOB.
No rejectable defects were identified. Licensee engineers performed a visual inspection of the Units 2 and 3, letdown piping without disturbing the existing insulation.
No damage was observed in either unit on insulation, hangers, piping or any visible component.
The NRC inspectors questioned licensee engineers about the need to perform visual inspections of the letdown piping with insulation removed to identify any deficiencies that may not be readily visible without removing insulation. Licensee engineers were concerned with radiation exposure that would be expended during further inspection of Units 2 and 3 letdown piping. The engineers subsequently selected piping analysis high stress areas for the Unit 2 letdown piping for removal of insulation and performance of further inspection.
Insulation was removed from the selected Unit 2 letdown line high stress areas.
The high stress areas included seven welds. The selected high stress area piping and welds were visually inspected by licehsee engineers and inservice inspection personnel.
No cracking or initiation points for cracking were visually observed.
The inspection expended 22 millirems. Licensee engineers concluded that no further inspection of Unit 2 or 3 was required.
Conclusions The inspectors concluded that the failed letdown line and Pipe Supports 13-CH-037-H-OOA and 13-CH-037-H-OOB met ASME Code design requirements.
The inspectors concluded that the design of Pipe Support 13-CH-037-H-OOB was poor due to the design's moment arm.
The inspectors concluded that licensee engineers performed appropriate evaluations of the failed letdown line to bound the conditions during the event and to allow an interim repair of the piping.
En ineerin Modification of Letdown Pi in Ins ection Sco e 93702 The inspectors reviewed and discussed with licensee engineers the design modification for the Unit 1 letdown line.
Observations and Findin s Licensee engineers modified the design of the Unit 1 letdown line by replacing the cutout section of damaged piping with a heavier walled 2-inch diameter, Schedule 80, approximately 18-inches long, piping of the same material specifications as the original pipe. The modification utilized two socket welded couplings to attach the new pipe section to the existing letdown line. The modification redesigned Pipe Supports 13-CH-037-H-OOA and 13-CH-037-H-OOB by eliminating the spring can support and welding stainless steel lugs to the new piping section.
The new supports restrained the new piping in both the axial and horizontal directions by means of structural steel members.
Licensee engineers superimposed the postulated loads, experienced during the letdown
h t
-13-line transient, on to the required design loads.
The piping analysis and pipe support calculations were performed for the new design and documented in stress Calculation 01-MC-CH-507, "Chemical and Volume Control System, Letdown Line to Letdown Heat Exchanger," Revision 3. Deficiency Work Order 00841404 was issued to perform the modification. The new design met applicable ASME Code requirements.
C.
'onclusions The inspectors concluded that the failed letdown line and Pipe Supports 13-CH-037-H-OOA and 13-CH-037-H-OOB were appropriately redesigned.
E1.3 E ui ment Root Cause of Failure Anal sis Ins ection Sco e 93702 The inspectors reviewed and discussed with licensee engineers the ongoing licensee activities for the equipment root cause of failure analysis.
The activities included evaluation of the letdown line thermal and hydraulic characteristics, diagnostic testing of letdown system valves, metallurgical examination of the failed piping, and review of past letdown line operating experiences.
Observations and Findin s Thermal H draulic Characteristics The inspectors reviewed Unit 1 plant computer data associated with the event and discussed the data with licensee engineers.
The inspectors noted that immediately prior to the event, the surveillance testing of Charging Pump E had been completed and only Charging Pump E was in servfce.
Letdown flowwas being maintained between 25 to 35 gpm. Letdown heat exchanger backpressure was being maintained at approximately 450 psig. Operators started the Charging Pump A to restore the unit to the normal two-pump lineup. An expected rapid decrease of the regenerative heat exchanger outlet temperature from 330'F to 180'F occurred.
Within a minute of the starting of the second charging pump, operators increased letdown flowand noted that letdown backpressure began to rapidly increase and oscillate. The flowrate had only increased about 4 gpm when pressure in the letdown system increased to approximately 600 psig and caused Relief Valve 1JCHNPSV0345 to lift. Operators took manual control of backpressure Control Valve 1JCHEPV0201P and set a 50 percent demand signal on the valve controller. This caused the pressure oscillations to dampen and the backpressure control valve was returned to automatic.
Two minutes later the pressure oscillations recurred.
Licensee engineers evaluated the data, noted that letdown backpressure was approximately 450 psig when a 200 psig pressure spike occurred when letdown flow was increased.
The pressure spike was sufficient to cause the relief valve to lift.
Licensee engineers considered that the relief valve liftcompounded the occurrence of pressure oscillations'in the letdown piping. Licensee engineers indicated that the letdown flow control valve and letdown backpressure control valve were not able to react
'
-14-fast enough to the changes in pressure when the relief valve lifted. Consequently, licensee engineers recommended a change to operating procedures to require letdown backpressure be minimized to approximately 200 psig prior to increasing the number of operating charging pumps and increasing letdown flow. At the conclusion of this inspection, licensee engineers were in the process of performing further thermal hydraulic analysis of the event in support of the equipment root cause of failure analysis.
Licensee engineers performed a 10 CFR 50.59 evaluation for an operating procedure change that would limitinitial backpressure for letdown restoration during start of a second charging pump.
During review, the engineer noted that Updated Final Safety Analysis Report, Chapter 15.6.2, included an analysis for a double ended letdown line break that assumed that certain control room indications/alarms would alert and enable operators to recognize the design basis break and isolate letdown within 10 minutes.
The engineer determined that one of those alarms, letdown flow hi/lo alarm, had not been installed in any of three Palo Verde units. Condition Report/Disposition Request 1-8-0368 was written.
10 CFR 50.59, "Changes, Tests and Experiments," states that the holder of a license authorizing operation of a production facilitymay make changes in the facilityas described in the safety analysis report, without prior Commission approval, unless the proposed change involves an unreviewed safety question.
Palo Verde Nuclear Generating Station Updated Final Safety Analysis Report, Chapter 15.6.2, "Double Ended Break of a Letdown Line Outside Containment,"
discusses the analysis of a letdown tine break event.
Section 15.6.2.2 states that the analysis assumes that the operator isolates the letdown line 10 minutes after the first three alarms resulting from the double ended break of the letdown line outsi4e containment upstream of the letdown line control valve, thereby, terminating any further release of primary flowto the auxiliary building. Table 15.6.2-1 lists the alarms that will be actuated for the double ended break of the letdown line outside containment event.
The table lists a letdown line low flow alarm as one of the first three alarms that willbe actuated for the event.
The licensee identified the condition that the letdown line low flow alarm had never been installed in any of the three Palo Verde Nuclear Generating Station units. However, no prior Commission approval had been obtained for the change, the change had not previously been evaluated as a change that did not involve an unreviewed safety question, and a change to the description in the Updated Final Safety Analysis Report had not previously been submitted to the Commission.
This was a violation of 10 CFR 50.59; however, the violation was of low-safety significance because eight other alarms would provide indication of a letdown line break to the control room operators.
A
'condition report/disposition request was written, but corrective actions had not yet been specified at the conclusion of the onsite inspection.
The inspectors noted that licensee corrective actions would be limited to either correcting the Updated Final Safety Analysis Report to reflect the plant and the analysis assumptions, or by installing the Updated Final Safety Analysis Report described alarm. Licensee personnel ackhowledged that the Updated Final Safety Analysis Report would be made correct by either changing the
II f
I
-15-report or by installing the described alarm. This nonrepetitive, licensee-identified and corrected violation is being treated as a noncited violation pursuant to Section VII.B.1 of the NRC Enforcement Policy'(50-528, -529, -530/9815-01).
Dia nostic Testin of Letdown Valves Diagnostic testing of the letdown backpressure control valves and letdown control valves were performed.
The relief valve was replaced and the existing valve was tested.
The diagnostic testing identified various differences between the as found valve settings and vendor recommendations.
Specifically, for the backpressure control valve, the initial current signal to open was high and the stroke length was slightly short.
Licensee engineers evaluated the observed difference in relation to the event and determined that the observed differences were not a contributing factor to the event but warranted further investigation as part of the ongoing root cause evaluation.
Metallur ical Examination of the Failed Pi in Licensee engineers sent the failed letdown pipe section to Southwest Research for metallurgical examination and determination of the cause of the crack. A licensee metallurgist was also sent to Southwest Research to participate in the examination of the pipe. At the conclusion of the inspection, the final results of the metallurgical examination were not available for review.
Review of Past Letdown'Line 0 eratin Ex eriences.
Licensee engineers reviewed Palo Verde and other industry experiences involving letdown pressure oscillations as part of its incident investigation. The review included a Korean plant experience and a 1992 Combustion Engineering study performed for Palo Verde. The incident investigation determined that the Combustion'Engineering recommended backpressure control valve controller setpoint changes had been
'accomplished.
The incident investigation concluded that previous operating history did not reflect a failure of the magnitude experienced during the event but problems with the control valves of the letdown system had been previously identified, evaluated, and corrected.
The incident investigation assigned additional review of past experiences as part of the equipment root cause of failure analysis.
E ui ment Root Cause of Failure Anal sis Re ort Through the course of the inspection, the inspectors discussed with licensee engineers the ongoing equipment root cause of failure analysis.
The inspectors observed that licensee engineers were performing comprehensive, albeit lengthy, evaluations of numerous facets of the event. As information became available, the technical aspects of the evaluation were'discussed with the engineers.
The NRC inspectors noted that information changed as evaluations continued.
For example, preliminary metallurgical results indicated a reverse bending failure mode.
However, subsequent discussions indicated that a torsional failure mode was being considered.
The NRC inspectors did not receive the completed equipment root cause of failure analysis report during the
Qt I
I
-16-inspection.
Further inspection to review the report, ascertain that appropriate aspects of the event had been considered, and to ascertain that appropriate corrective actions had been performed to address conditions associated with the event is an inspection followup item (50-528/9815-02).
c.
Conclusions The licensee-identified, low-safety significant, failure to install the Updated Final Safety Analysis Report, Chapter 15.6.2, described letdown flow hi/lo alarm in all three Palo Verde units was identified as a noncited violation.
The inspectors concluded that licensee engineers were performing comprehensive, albeit lengthy, evaluations of numerous facets of the event to determine the equipment root cause of failure.
The NRC inspectors did not receive the completed equipment root cause of failure analysis report during the inspection.
Further inspection to review the report, ascertain that appropriate aspects of the event had been considered, and to ensure that appropriate corrective actions had been performed to address conditions associated with the event is warranted.
X1 Exit Meeting Summary The inspector presented the inspection results to members of licensee management at the conclusion of the inspection by means of a telephone exit meeting on August 6, 1998. The licensee personnel acknowledged the results of the inspection.
The inspector asked licensee management and staff whether any material examined during the inspection contained proprietary information. No proprietary information was identified. No proprietary information is included in this repor Licensee ATTACHMENT SUPPLEMENTAL INFORMATION PARTIALLIST OF PERSONS CONTACTED D. Garnes, Department Leader, Operations
D. Fan, Department Leader, System Engineering R. Fullmer, Director, Nuclear Assurance K. Graham, System Engineer, Charging System W. Ide, Vice President, Engineering A. Krainik, Department Leader, Regulatory Affairs.
J. Levine, Senior Vice President, Nuclear D. Mauldin, Director, Maintenance G. Overbeck, Vice President, Nuclear Production K. Parrish, Section Leader, Transient Analysis M. Radspinner, Section Leader, Mechanical-NSSS, Design D. Smith, Director, Operations NRC J. Moorman, Senior Resident Inspector N. Salgado, Resident Inspector 93702 INSPECTION PROCEDURES USED Prompt Onsite Response to Events at Operating Reactors
~ ITEMS OPENED
~oened 50-528/9815-01 50-529/9815-01 50-530/9815-01 50-528/9815-02 NCV Failure to Install Letdown Flow Hi/LoAlarm NCV Failure to Install Letdown Flow Hi/LoAlarm NCV Failure to Install Letdown Flow Hi/Lo Alarm IFI Review of Letdown Line Equipment Root Cause of Failure Report DOCUMENTS AND PROCEDURES REVIEWED Procedures 40ST-9CH06, "Charging Pump Operability Test," Revision 3
I
-2-41OP-1CH01, "Charging and Volume Control System Normal Operations," Revision 28 40AO-9ZZ05, "Loss of Letdown," Revision 3 Condition Re ort/Dis osition Re uest 1-8-0322 1-8-0335 Other Documents Work Order 00841404, "Remove and Replace Failed Unit 1 Letdown Piping" Memorandum 162-08369-KCP/AMT, "NFM Nuclear Safety Assessment:
Unit 1 Letdown Break on May 21, 1998," issued May 21, 1998 (erroneously dated March 21, 1998)
Memorandum 162-08370-KCP/HAT, "Safety Assessment of Unit 1 Letdown Line Isolation,"
dated May 21, 1998
"Letdown Restoration Gameplan" document, dated May 22, 1998 Units 1, 2, and 3, Night Order, dated May20, 1998 Unit 1 Letdown Line Failure Incident Investigation Report, dated July 1, 1998 Unit 1 Letdown Line Weld Failure Engineering Evaluation Report, dated June 26, 1998 Drawing 01-P-CHF-201, "AuxiliaryBuilding Isometric, Chemical and Volume Control System, Letdown Line to Letdown Heat Exchanger," Revision 3 Drawing 01-M-CHP-001, P & I Diagram, "Chemical and Volume Control System," Revision 17 Drawing 13-CH-037-H-OOA, "Pipe Support Assembly," Revision 2 Drawing 13-CH-037-H-OOB, "Pipe Support Assembly," Revision
Stress Calculation 01-MC-CH-507, "Chemical and Volume Control System, Letdown Line to Letdown Heat Exchanger," Revision 0
-