IR 05000528/1993025
| ML20045C592 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 06/07/1993 |
| From: | Wong H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML20045C587 | List: |
| References | |
| 50-528-93-25, 50-529-93-25, 50-530-93-25, NUDOCS 9306240045 | |
| Download: ML20045C592 (60) | |
Text
a
.
U. S. NUCLEAR REGULATORY COMMISSION
REGION V
Report Nos.
50-528/93-25, 50-529/93-25, and 50-530/93-25 Docket Nos.
50-528, 50-529, and 50-530 License Nos.
NPF-41, NPF-51, and NPF-74 Licensee:
Arizona Public Service Company
-
P. O. Box 53999, Station 9082 Phoenix, AZ 85072-3999 Facility Name:
Palo Verde Nuclear Generating Station Units 1, 2, and 3 Meeting at:
Region V Office, Walnut Creek, California Prepared by:
B. J. Olson, Project Inspector
,
h 0/N9J Approved by:
r HT Wong, Chief (/
Date Signed
'
Reactor Projects S6ction II Summary:
A management meeting was held on May 24, 1993, to discuss: (1) the findings of the NRC Augmented Inspection Team which reviewed the licensee's response to the Unit 2 steam generator tube rupture event of March 14,1993,(2) the APS efforts to determine the root cause of the steam gene'rator tube rupture, (3) the APS assessment of reactor trip breaker problems, and (4) other topics of mutual interest. The meeting agenda and a copy of the slides used in the licensee's presentation are enclosed.
.
l t
,
9306240045 930608 PDR ADOCK 05000528'
'
O PDR
..
.
. _
_
.
_
_
_
__ _
-.
.
DETAILS 1.
Meetino Attendees Arizona Public Service Company W. Conway, Executive Vice President, Nuclear E. Simpson, Vice President, Nuclear Engineering & Projects
-
J. Levine, Vice President, Nuclear Production R. Stevens, Director, Nuclear Regulatory & Industrial Affairs J. Bailey, Director, Site Technical Support R. Adney, Unit 3 Plant Manager R. Schaller, Unit 1 Assistant Plant Manager P. Guay, Unit 3 Chemistry Manager K. Sweeney, Senior Project Engineer, Nuclear Engineering Department Nuclear Reculatory Commission K. Perkins, Director, Division of Reactor Safety and Projects S. Richards, Deputy Director, Division of Reactor Safety and Projects i
C. VanDenburgh, Chief, Reactor Projects Branch
'
H. Wong, Chief, Reactor ProjectsSection II
'
P. Morrill, Chief, Operations Section D. Kirsch, Technical Assistant R. Pate, Chief, Safeguards, Emergency Preparedness, Non-Power Reactor Branch C. Trammell, Senior Project Manager, NRR W. Ang, Chief, Engineering Section J. Sloan, Senior Resident Inspector
'
A. MacDougall, Resident Inspector C. Myers, Reactor Inspector L. Coblentz, Senior Radiation Specialist B. Olson, Project Inspector Others M. Benac, Manager, El Paso Electric R. Henry, Site Representative, Salt River Project M. Short, Manager, Site Technical Services, Southern California Edison
D. Summers, Public Service Company New Mexico 2.
Details Mr. Perkins provided opening remarks for the NRC and indicated that the
meeting was one of several routine management meetings conducted with Region V licensees. He also indicated that the meeting provided a good opportunity to review several items of mutual interest, and he summarized the agenda for the meeting. Mr. Conway provided opening remarks for Arizona Public Service Company (APS) and indicated that he and his staff were prepared to answer questions about the issues to be discussed.
l
-
.
--.
..
.
,
..
_
_
__
_
-
.__
_
.
a.
Augmented inspection Team The NRC initiated an Augmented Inspection Team (AIT) to review the events surrounding the Unit 2 steam generator tube rupture which occurred on March 14, 1993. The findings of the AIT are documented in NRC Inspection Report 50-529/93-14.
Mr. Adney provided APS comments to the AIT report. APS generally agreed with the AIT's findings, and Mr. Adney discussed issues pertaining to:
(1) the radiation monitoring system, (2) training, (3) Emergency Operating Procedures, and (4) items regarding general operating procedures and secondary water inventory management. After reviewing
'
each issue, Mr. Adney described the actions taken or proposed by APS to correct observed deficiencies.
NRC questions focussed on the APS evaluation of each issue.
.
Mr. Richards made the observation that the actions to respond to the
' '
event were, in part, driven by a radiation monitor (condenser exhaust monitor) with an inaccurate setpoint that did not alarm as anticipated.
Mr. Adney agreed with the observation and indicated that APS was reviewing other scenarios where indications from a single instrument might drive decisions and actions.
b.
Unit 2 Steam Generator Tube Inspections
Mr. Schaller reviewed the APS efforts to determine the root cause of the tube rupture which occurred in Unit 2 steam generator 2 on March 14, 1993. Mr. Schaller provided an overview of the experience and qualifications of the root cause of failure team members and reviewed a chart of potential failure modes.
He also provided the status of eddy current inspection efforts and the preliminary results from chemical analyses performed on tubes pulled from steam generator 2.
Mr. Schaller described the examination of the steam generator's operating history and
fabrication records. He also described some of the characteristics of the observed eddy current results and explained how the results were affected by the method used to fabricate the steam generator tubes. At the time of the meeting only preliminary metallurgical results were available, and APS did not have a probable root cause for the tube
!
rupture.
Mr. Trammell suggested that a working meeting should be scheduled in which APS could share their information with the technical experts from the NRC.
APS officials agreed that a working meeting to discuss steam i
generator issues would be beneficial. Subsequently, a meeting was i
scheduled for June 3,1993.
i c.
Reactor Trio Breakers Mr. Bailey reviewed the history of reactor trip breaker problems f
beginning with a 1992 event in which a Westinghouse trip breaker failed to fully open during testing.
In that instance, the breaker failure was
_
_
_
_
__
l I
i
-
i
J partially attributed to an inadequate maintenance program, and program changes were made.
In 1993, a Westinghouse breaker failed to open while racked out in the test position, but Mr. Bailey explained that the cause of the failure, a deformed spring return mechanism, would not have prevented the breaker from opening when racked in to the operating position. No other significant Westinghouse reactor trip breaker problems had been observed at Palo Verde.
Mr. Bailey indicated that the majority of the reactor trip breaker problems involved General Electric breakers, and in each instance the breaker would have opened if required. Mr. Bailey went on to describe the APS efforts to determine to root causes of the General Electric breaker problems, and he stated that APS now has a good understanding of the breaker's operation and behavior.
Nevertheless, Mr. Bailey stated that all Palo Verde reactor trip breakers will be replaced with a different Westinghouse breaker (model DS-416) that offers proven reliability and additional opening forces.
d.
Unit 2 Snubber Failures Mr. Bailey described the APS efforts tc, determine the cause of surveillance test failures of mechanical piping snubbers during the current Unit 2 refueling outage. The majority of test failures occurred in mechanical snubbers categorized as small in size (1/4 and 1/2 kip),
and five different failure mechanisms were determined for the category.
As a result of the failures, APS was required by Technical Specifications to test the majority of the snubbers in Unit 2.
Stress analyses and inspections performed as a result of the test failures did not reveal adverse effects on piping systems. Mr. Bailey also described the basis for the APS conclusion that the high failure rate in Unit 2 did not affect the operability of snubbers in Units 1 and 3.
Mr. Wong asked if APS would perform any design changes to upgrade the snubbers in Unit 2.
Mr. Bailey replied that APS will work to eliminate snubbers from the Unit since analysis demonstrated that allowable pipe
,
stresses would not be exceeded even with the snubbers in a " locked up"
"
condition.
e.
Steam Bypass Control Valve Inadvertent Openina Mr. Bailey briefly reviewed an April 2,1993, event in which the Unit 3
!
steam bypass control valves inadvertently opened during a post-
'
modification test.
The plant response to the event was determined to be within analyzed limits.
APS determined that a prior plant modification added an operation feature to the valve master controller which had not been recognized, and this feature caused the valves to open on April 2.
APS concluded that the earlier plant modification had not been adequately reviewed or tested. Changes were made to the plant modification program as a result of this event.
..
_
_...
_
_
_. -
-
.
_. _..
_
_
_ _ _
_ _
k
.
f.
Engineer Qualification Mr. Simpson briefly provided information regarding the qualifications of
,
engineers.
He indicated that APS upgraded and standardized the descriptions of job positions in 1990, but found that the qualifications of some personnel did not meet the standardized position descriptions.
As a result, APS is rewriting and reclassifying some job descriptions.
Mr. Simpson also indicated that the Updated Final Safety Analysis Report had been recently revised Eto state that personnel would be qualified to the requirements of ANSI 3.1, rather than the requirements of ANSI 3.1 and the job position description.
'
g.
Closina Remarks Mr. Conway stated that he and his staff came prepared to deliver a large amount of information and added that he hoped the presentations were useful. Mr. Perkins indicated that the discussions were beneficial and expressed his appreciation to the APS personnel.
He stated that the NRC will need to stay in close communication with APS as progress continues in the evaluation of the Unit 2 steam generator tube inspection results.
Mr. Perkins added that the NRC will need to understand the APS basis for conclusions leading to the restart of Unit 2.
Mr. Conway replied that Mr. Stevens will be coordinating the Unit 2 restart plan, and APS
'
intends for the NRC to have a complete understanding of the restart actions.
The status of corrective actions for licensed operator requalification examinations was not discussed due to time constraints.
'
,
l
.
'
-.-
- - -
- - - -...
-
-
--
_
-
.
__
._
_
.
..
. _ _.
.
.
,
APS/NRC MANAGEMENT MEETING q
MAY 24, 1993-j AGENDA
,
.
I.
OPENING REMARKS K. E. PERKINS/
-
W. F. CONWAY II.
NRC/APS AIT/IIT STEAM GENERATOR D. F. KIRSCH /
'
TUBE RUPTURE EVENT RESULTS R. J. ADNEY-
I III.
UNIT 2 STEAM GENERATOR R. F. SCHALLER TUBE INSPECTION STATUS
.
IV.
REACTOR TRIP BREAKER STATUS J. A. BAILEY
'
V.
UNIT 2 SNUBBER STATUS J. A. BAILEY
,
VI.
UNIT 3 SBCV TRANSIENT J. A. BAILEY
-
'
VII.
REQUALIFICATION EXAMINATION J. M. LEVINE CORRECTIVE ACTION STATUS VIII.
ENGINEER QUALIFICATION E. C. SIMPSON IX.
CLOSING REMARKS W. F. CONWAY/
K. E. PERKINS
.
.
.
- -.
.
-
.
-
-
..,-a,,..
s.
u a
e
. ~ -.
-.
.
as
,
..
-
-- - -
.
.
.
.
.
,
I NRC/APS AIT/IIT STEAM GENERATOR TUBE RUPTURE EVENT RESULTS
,
?
o RADIATION MONITORING SYSTEM (RMS) ISSUES o
TRAINING ISSUES i
!
o EMERGENCY OPERATING PROCEDURE (E0P) ISSUES
<
l o
ADDITIONAL ISSUES RJA-1 05/24/93
_.. _ _. -.
,
.
-
-
- - -
--.
-
-
. - -
.
..
..
-.
.
-
-.
.
!
'
,
RMS ISSUES i
PRIOR TO EVENT, MISSED OPPORTUNITY T0. IDENTIFY-e CONDENSER VACUUM EXHAUST RADIATION MONITOR (RU-141).
PROBLEM
,
.
CORRECTIVE ACTION:
.
1.
REVIEWED DATA FROM LOW RANGE EFFLUENT MONITORS (RU-141,143,145) AND CONTAINMENT
ATMOSPHERE MONITOR FROM UNITS 1 AND 3 a
2.
PROGRAM TO TREND RMS DATA FOR EFFLUENT MONITORS
,
3.
INDUSTRY EVENTS TRAINING
i
RJA-2-05/24/93
..
..
.
-.
_,..
.
,.
.
. - - -
,
RMS ISSUES (CONT'D1
.
ALARM SETPOINT CHANGED WITHOUT REQUIRED APPROVAL CORRECTIVE ACTION:
1.
REVIEWED DATA FROM UNITS 1 AND 3 2.
MANAGEMENT MEETINGS WITH EMPLOYEES 3.
INDUSTRY EVENTS TRAINING
'
4.
ENHANCING RMS SETPOINT CHANGE PROCESS RJA-3 05/24/93
.-
RMS ISSUES (CONT'D)
ALARM RESPONSE PROCEDURE REQUIREMENTS NOT FULLY o
IMPLEMENTED EVALUATION:
1.
TECHNICIAN ACTIONS WERE APPROPRIATE.
2.
METHOD USED TO VALIDATE ALARM NOT IN COMPLIANCE WITH PROCEDURE.
3.
DATA FROM RU-141 USED TO VALIDATE ALARM INSTEAD OF REVIEWING RU-140 DATA.
4.
ALARM RESPONSE PROCEDURE IS BASED ON RESPONSE TO SINGLE ALARMS, NOT MULTIPLE ALARMS.
CORRECTIVE ACTION:
REVISING RMS ALARM RESPONSE PROCEDURE T0:
1.
PROVIDE MORE INTEGRATED APPROACH TO ALARM ASSESSMENT.
2.
ALLOW TECHNICIAN FLEXIBILITY IN PRIORITIZING SELECTION OF ALARMS TO BE ASSESSED.
i i
RJA-4 05/24/93
...
-
-
- -
- i
,
.
RMS ISSUES (CONT'D)
ALERT AND ALARM SETP0INTS APPEAR TO BE BASED ON OFF-e SITE DOSE LIMITS RATHER THAN ABILITY TO PROVIDE INDICATION OF A SGTR EVENT.
RU-139 AND RU-140 MAIN STEAM LINE RADIATION
-
MONITOR SETPOINTS ARE BASED ON BACKGROUND, NOT OFF-SITE DOSE LIMITS EVALUATION:
1.
EVALUATE MONITOR RESPONSE AND USE RU-141, CONDENSER VACUUM EXHAUST RADIATION
-
MONITOR SETP0INTS ARE BASED ON OFF-SITE DOSE LIMITS CORRECTIVE ACTION:
1.
RE-EVALUATED RESPONSE TO INFORMATION NOTICES 2.
DESIGN CHANGE FOR CONDENSER EXHAUST PATHWAY 3.
LOWER ALERT SETPOINT IN UNITS 1 AND 3 RJA-5 05/24/93
..
.
.
-
,
RMS ISSUES (CONT'D)
- RMS SELF ASSESSMENT
PURPOSE:
SELF IDENTIFY AND CORRECT ANY PROBLEMS, NOT ONLY IDENTIFIED IN UNIT 2 EVENT, BUT, THROUGHOUT THE RMS
"
PROGRAM
.
SCOPE:
PROCEDURES, TRAINING, UNIT CONSISTENCY, AND EMPLOYEE PERCEPTIONS MULTIDISCIPLINE TEAM
,
INITIAL FINDINGS / RECOMMENDATIONS:
,
'
1.
PROCEDURE COMPLIANCE
'
2.
PROCEDURE ENHANCEMENTS 3.
CHEMISTRY PERSONNEL TO BE PART OF QA AUDIT TEAMS 4.
ADDITIONAL TRAINING BASED ON FEEDBACK SUCH AS:
PLANT SYSTEM INTERFACE
-
CORE DAMAGE ASSESSMENT
-
RJA-6 05/24/93
. -
-
- -.
.
.
TRAINING ISSUES o
RADIATION MONITOR ALARM RESPONSE TIME FOR UNIT 2 SGTR EVENT DIFFERENT THAN SIMULATOR-SGTR SCENARIO CORRECTIVE ACTION:
1.
BRIEFED LICENSED OPERATORS ON EVENT 2.
DEVELOP SIMULATOR SCENARIO AND TRAINING BASED ON UNIT 2 EVENT INCLUDING DIAGNOSIS OF TUBE RUPTURE RJA-7 05/24/93
'
.
.
...
.
TRAINING ISSUES (CONT'D1
,
o SMALL STEAM RELEASE PATHWAYS, SUCH AS AUXILIARY STEAM RELIEF LINE, NOT ADEQUATELY ADDRESSED IN TRAINING CORRECTIVE ACTION:
1.
DEVELOP TRAINING ON OTHER STEAM RELEASE PATHWAYS 2.
REVISE PROCEDURES TO LIST OTHER STEAM RELEASE PATHWAYS RJA-8 05/24/93
--
.
.
. -
.
.
-
.
.
TRAINING ISSUES (CONT'D)
SIMULATOR INDICATES FULL RANGE OF HPSI FLOW, WHILE
PLANT CONTROL BOARD INDICATES ZER0 FLOW WHEN FLOW-IS
'
LESS THAN TEN PERCENT CORRECTIVE ACTION:
1.
BRIEFED LICENSED OPERATORS 2.
SIMULATOR HPSI FLOW RESPONSE CORRECTED TO SIMULATE PLANT CONTROL BOARD INDICATION 3.
OTHER FOXBOR0 FLOW INDICATORS WERE EVALUATED AND CORRECTED TO SIMULATE PLANT CONTROL BOARD INDICATION 4.
EVALUATE PLANT HPSI FLOW INDICATION
,
RJA-9 05/24/93
'
.
.
E0P ISSUES e
E0P DIAGNOSIS OF SGTR CAUSE OF DELAYED DIAGNOSIS:
1.
DIAGNOSTIC SCHEME USED SNAPSHOT PHILOSOPHY 2.
ACTUAL RMS RESPONSE DIFFERENT THAN EXPECTED 3.
STEAMLINE MONITOR CLEARED, BLOWDOWN MONITORS ISOLATED, AND CONDENSER EXHAUST MONITOR EQUIPMENT FAILURE 4.
FUNCTIONAL RECOVERY PROCEDURE STEP FOR DIAGNOSING SGTR WAS NOT CONTINU0USLY APPLICABLE CORRECTIVE ACTION:
1.
REVISED DIAGNOSTIC LOGIC FLOWCHART i
2.
REVISE PROCEDURES
.
,
3.
VALIDATION AND VERIFICATION PROCESS REVIEW RJA-10 05/24/93
.
.
.
ADDITIONAL ISSUES o
E0P EXIT TO GENERAL OPERATING PROCEDURE WAS DELAYED WHILE EXTENSIVE VARIANCE WAS PREPARED CORRECTIVE ACTION:
1.
DEVELOP METHODOLOGY TO TRANSITION FROM E0Ps e
VCT OVERFILL CORRECTIVE ACTION 1.
REVISE PROCEDURE TO CONTROL VCT LEVEL
,
CHARGING PUMP TRIP ON LOW NPSH WHEN BAMP IS IN o
RECIRCULATION MODE CORRECTIVE ACTION:
1.
REVISED PROCEDURES 2.
EVALUATE ALTERNATE FLOW PATHS 3.
REVISE PROCEDURES BASED ON EVALUATION RJA-11 05/24/93
'
.
,
-
.
ADDITIONAL ISSUES (CONT'D)
WATER MANAGEMENT PROBLEM FOLLOWING UNIT 2 EVENT CORRECTIVE ACTION:
ESTABLISH WATER MANAGEMENT PLAN T0:
-
1.
MINIMIZE NEW INVENTORY TO CONDENSER 2.
PROVIDE STRATEGY FOR STORAGE AND MONITORING 0F INCREASED INVENTORY 3.
PROVIDE STRATEGY / METHODS TO MINIMIZE CONTAMINATION 0F STORAGE FACILITIES 4.
PROVIDE STRATEGY / METHODS TO PROCESS INCREASED INVENTORY
.
RJA-12 05/24/93
FA 1.;
,-4
-
.
ab
.-
,
,y'
A
>
- i t
. :
r
!
,
,
i
)
I
PVNGS UNIT TWO S/G TUBE RUITURE ROOT CAUSE OF FAILURE r
I
' l PRESENTED BY j
l RICIIARD SCIIALLER l
u i
I l
l
_
__. _. _.,
_.
-
. _,..
--,_. -
..
.
. _
.
..
.
..
-
.
t
i PVNGS UNIT TWO S/G TUBE RUPTURE ROOT CAUSE OF FAILURE
,
,
'
I. RCF EFFORT A. STEAM GENERATOR WORKING rkOUP / ROOT CAUSE OF FAILURE TEAM l
B. POTENTIAL SCENARIO DEVELOPMENT l
,
C. RCF ANALYSIS FLOWCIIART
.
,
,
II. CURRENT STATUS OF INDICATIONS i
,
IIL ONGOING INVESTIGATIVE ACTIVrIY
_
IV. TUBE PULLING AND METALLURGY TEST RESULTS l
l s
'
.
i l
_
.. _ _ _
,
I. RCF EFFORT A. STEAM GENERATOR WORKING GROUP / ROOT CAUSE OF FAILURE TEAM 1.APS R. SCHALLER (PLANT)
M. IIODGE (ENGR)
A. MORROW (ISI)
- D. IIAUTALA (QUAL ENGR)
- C. BRECIITEL (SRO/STA)
- M. RADSPINNER (ENGR)
'
- S. QUAN (SRO/STA)
- L JOIINSON (CIIEM)
K. SWEENEY (ENGR)
2. INDUSTRY M. SNIEGOWSKI (CE PROJELT REPRESENTATIVE)
- R. BOBERG (BWNS MANAGER OF PROJECT ENGINEERING)
- W. IIEILKER (CE MANAGER-ANALYTICAL ENGINEERING DEPT)
P. PAINE (EPRI MANAGER-CORROSION & MATS FOR S/G MGMNT PROJ)
S. BROWN (EPRI EDDY CURRENT CONSULTANT)
DR, SRI KANTIAII (EPRI MANAGER-S/G MECIIANICAL) FOR S/G PROJ)
DR. J. BEGLEY (ODONNELL ASSOC. METALLURGY EXPERT)
DR. C. CIIIU (RCF/2-PIIASE ANALYSIS)
'
DR. S. CIIEN (FLOW INDUCED VIBRATION ARGONNE NATL 1AB)
DR. P. GRIITITII ( MIT THREMAL IIYDRAULICS MODELLING)
- RCF TEAM MEMBER B. POTENTIAL SCENARIO DEVELOPMENT
- ODSCC (ICSCC/ IGA)
- FATIGUE
- TIIERMAL EXPANSION
- 2-PIIASE FLOW INDUCED VIBRATION
- FRITTING C. RCF ANALYSIS FLOWCIIART
~!
"
i
--
J I
,
IL CURRENT STATUS OF INDICATIONS A.100% BOBBIN INSPECTION OF #1 AND #2 S/G COMPLETE B. #1 S/G
- J2 TUBES WITIl AXIAL INDICATIONS IN #1 S/G
- 7 MIDSPAN
- J AT SUPPORTS OSII,0911, BW1
- 2 CAPTURED BY THE TUBESIIEET
- 66 TUDES EVALUATED AS NEEDING PLUGGING IN #1 S/G C. #2 S/G
- 32 TUBES %TTII AXIAL INDICATIONS IN #2 S/G
- 13 MIDSPAN
- 15 AT SUPPORTS 071I,0811,091I, BW1
- 4 AT 0111 SUPPORT (INCLUDES 1 FROM 2R3)
- 132 TUBES EVALUATED AS NEEDING PLUGGING IN #2 S/G
-.
.
.
..
-
._.
-
-.
-
,
.
.
.
'
.
!
.
III. ONGOING INVESTIGATIVE ACTITTIY -
A. CIIEMISTRY REVIDV 1. LEAD ANALYSIS
2. RESIN INTRUSION
!
3. CAUSTIC CREVICE ENVIRONMENT
!
B. OPERATIONS ANALYSIS - LEVEL OSCILLATIONS
,
C. FABRICATION REVIDV D. 'nVO PIIASE FLOW INDUCED VIBRATION ANALYSIS E. MRPC INSPEC1' IONS
'
1. INITIAL SCOPE 2. EXPANSIONS
,
,
- ROWS 116,117,118
- 150 TUBES AROUND R105-C156
- OSII BW1 PLUS 0111 AND TSII ON TUBES %TTII INDICATIONS i
- ARC OF INTEREST
- CIIECKERBOARD
- EXPANDED ARC 4. IN TUBES IN A CIIECKERBOARD PATTEIUi 5. SAMPLE SIZE STATISTICAL ANALYSIS F. ROTATING FIELD ECT PROBE
,
!
G. ECT TIIRESIIOLD EVALUATION AND PILGERING EFFECTS IL SECONDARY SIDE ENTRY
'
l. RESIN ON CAN DECK 2. NO EVIDENCE OF BINDING OR IlOWING
!
I. INDUSTRY EXPERIENCE REVIEW J. EPRI CONSULTATION
,
'
)
h
-
-
-
,,
A
-.
.
.
IV. TUBE PULLING AND METALLURGY TEST RESULTS i
A. LIST DEVELOPED (8 TUBES)
i 1. SELECTION CRITERIA
- RUI"TURED TUBE.
- AXIAL MIDSPAN INDICATIONS
- RPC VS. BOBBIN SENSITIVTIY
- 0111 CRACKS
"
- BURST TEST CANDIDATES
- CLEAN TUDE FOR TIIRESIIOLD VERIFICATION
- ANALYSIS OF DEPOSITS 2. CANDIDATE ATTRIBUTE BREAKDOWN
.
- 'IWO SIIORT TUBES
,
- FOUR TUBES MTTII MIDSPAN INDICATIONS
'
- TWO MTITI NO BOBBIN INDICATION
'
- ONE CLEAN TUBE WITII CIRCUMFERENTIAL INDICATION
- ONE %TTII DEPOSIT ECT INDICATIONS B. PRE-PULL TEST PROGRAM
- CRACK ORIENTATION / DEPOSIT ORIENTATION
- FULL LENGTII MRPC
- SPECIALIZED BOBBIN ELT
- BOWL RADIUS GAGING C. TUBE PULL ISSUES
'* METIIOD OF WIIIP CUT
- SECONDARY SIDE IIARVEST
- FAILURE TO CUT RUPTURED TUDE
,
.
D. VIDEO PROBE DATA E. WIIIP CUTTER EFFECT ON INDICATION F. METALLURGY RESULTS I
1. INITIAL VISUAL INSPECTION: WEAR INDICNTION AND DEPOSIT 2. METALLURGY TESTS SIIOW GRAPIIITE FIBERS IN DEPOSIT ON 117-40 3. PRESENCE OF MO, PB, S, CU, FE NOTED ON TUBE OD t
4. ELT ANALYSIS REVEALED TIIAT INDICATION ON 117-40 OPENED UP 5. IlWNS IDENTIFIES FAILURE MECIIANISM ON 117 40 AS IGA /lGSCC
,
6. CE BURST TEST OF 105156 IDENTIFIED IGA /ICSCC 7. PT/RT OF 116-41 SIIOWS NO CIRCUMFERENTIAL CRACK
.. - -
..,,,,,-,..ew..,.
m h
,
-
&
ROOT CAUSE INVESTIGATION TEAM --- SGTR EVENT FAILURE MODES I
r
- -=>
l l
I I
u-,.
~_~
a m
' fjppi: Aj l
l l
t
'
I l
I l
l l
l g
.y:rjy
.
_p
-
--
--
-
,,_
_,, -
--
-.--
.. -
.
g,y ;
,
I
,
I I
I I
l l
i@
.a
.a_
-
-.
~
_
_
<
mo
,
J%
- ur.
-4g6;
";_" '**
corhm based en examavation / testng of l
=
=
-
%'*Q puDed tubes
__
phdin -
lsp[ig a: aiysis seguired to support
-
- , :: g :.=
x-.;
.-
. -. -
.-
-
g-known to be susceptibleIfactor
'WMETE
"L"'
- "**""
~ * * '
no data has indicated problems in this area
--- -
l%sl'
..se
muc+g n.,,..~...,,,,,
...
- -
.-..
,
a.
~,_
,-
-.
. -
-
..
-
..,. -.
.
...
.
.-
.,,
. -....,.....,
...
-
.
..
.. -
- .
.
.
!
,
.
Unit 2 Snubber Status i
i h
!
t PVNGS PRESENTATION
. TO NRC REGION V May 24,1993
.
....
..
.
_
.
..
-
-
__.
-
.
Unit 2 Snubber Status Contents
.
I.
Overview
.
II.
Sununary of the Snubber ERCFA.
.
III.
Steam Supply to Aux Feed Water Terry Turbine Piping
,
IV.
Reactor Coolant System Drain Line RC-070/071 Piping V.
Small Snubber Failure Mechanisms VI.
Large Snubber Failure Mechanisms i
VII. Perspective On Test Criteria l
Unit 2 Snubber Status Overview
.
Due to a higher number of failures associated with the steam supply piping for
-
the "A" Auxiliary Feedwater Pump Terry Turbine, an Investigation Team was formed to perform an Equipment Root Cause for Failure Analysis on the snubbers in question and evaluate impact on Unit I and. Unit 3. As the failure rate increased in Unit 2, as a whole, the scope of the Task Team's investigation was broadened to include an evaluation of all failures found during the current Unit 2 outage.
Prior to the current Unit 2 outage only a single snubber had failed during
-
'
Surveillance Functional testing.
Data shows that although the current outage failure rate (9.4%) is above the
industry average (7.0%) the overall Unit 2 failure rate (6.9%) is consistent with the industry.
Failures have occurred predominately in small (1/4 and 1/2 kip) and large (35
and 100 kip) groups of mechanical snubbers. The medium (1,3, and 10 kip)
snubbers experienced only 2 failures.
I L
>
.
-
~
~
-
w
~
-.,,
.
Unit 2 Snubber Status Summary of the Snubber Testing ERCFA Timeline
-
-
March 18 - First snubber testing for current outage of Unit 2
-
March 31 - Scope of snubber testing was expanded to include all 1/4 &
1/2 kip snubbers April 12 - Scope of testing was expanded to include all snubbers on the
-
Steam supply lines to the AF Terry Turbine after a review of test data
,
showed a high incidence of failures of the small snubbers on those lines.
-
May 7 - Major conclusions for small snubbers, AFW failure mode group, and impact on Unit I and 3 completed. Large snubbers still working.
,
Emphasis was place on.the evaluation of operability of snubbers and the
transportability of problems found in Unit 2 to the other Units. All stress analysis were applicable to all three units.
No single mechanism has been identified for the snubber failures.
A stress analysis was performed, as required, on the piping system for each
snubber failure. In all cases but one, the piping stresses were below code allowable limits. For the single failed case, the follow up visual and PT
-
inspections were completed satisfactorily (no indication of weld or pipe support damage).
.
SNUBBER FAILURE SUMMARY FAILURE MECHANISMS TRANSIENT /
INSTALLATION /
MANUFACTURING /
SIZE
- TESTED
- FAILED VIBRATION ENVIRONMENT MISHANDLING /
DESIGN UNKNOWN MAINTENANCE -
SMALL 125
5
2
1
!
MEDIUM 124 2**
0
0
LARGE 71*
9'
0
2
- Testing is not yet complete. 2 35's awaiting completion of SG activities.
- * Disassembly of one snubber for investigation remains to be don.
.
Unit 2 Snubber Status Steam Supply to Aux Feed WaterTerry Turbine Piping Ten of the thirty snubbers on the Auxiliary Steam and Main Steam lines to the
Terry Turbine failed their functional test.
Inspection of the disassembled failed snubbers identitled four failure modes -
Transient, Environmental, Installation, and Unknown.
Piping stress analysis performed on the steam lines with the "As Found"
failure configuration (10 failed snubbers) were within code allowable stresses.
None of the failure modes for his group were considered to be common mode
failures. Therefore, it was not expected that they would occur in significant numbers on the same piping systems in Units 1 and 3. Ilowever, to ensure that a large number of failures would not adversely airect the steam lines in the other two units a " Worse Case" stress analysis was performed.
The " Worse Case" stress analysis modeled all 30 of the steam line snubbers
" locked-up". It identified one location in which code allowables were exceeded. Results of the " Worse Case" analysis showed that a large
percentage of the snubbers on these lines could be removed under the snubber -
)
reduction program, f
Prior to the current Unit 2 outage the Auxiliary Steam supply line to the Terry
)
Turbine (AF-031) was identified as a high failure rate system. In Units 1 and 3 this line receives additional testing. This line was not originally scheduled for additional testing in Unit 2 due to the upgrade of the snubbers on that line in Unit 2 during 1985.
.
-
&
Unit 2 Snubber Status
.
Unit 2 AFW Failure Mode Group Comparison With Same Snubbers in Other Units UNIT Group of Snubbers Within the AFW Steam Supply Piping to the Unit 1 Unit 2 Unit 3 Terry Turbine 12 of 16 tested and 1 of 16 tested and 8 of 16 tested during Main Steam Supply passed during each of passed prior to 2R4.
3R1 through 3R3.
Piping the first two outages.
16 of 16 tested with 4 1 failure during 3R3.
(16 snubbers)
A single failure during failures during 2R4.
1R3.
1 of 6 tested and 6 of 6 tested during 6 of 6 tested each of
-
Auxiliary Steam Supply the three outages.
passed prior to 2R4.
3R2 with 3 failures.
(6 snubbers)
2 failed during 1R1.
6 of 6 tested with 3 6 of 6 tested during Same 2 failed 1R3.
failures during 2R4.
3R3 with 1 failure.
9 of 9 tested and 2 of 9 tested and 1 tested and failed Steam Bypass Around passed during 1R1.
passed prior to 2R4.
during 3R1. Next test Main Steam Supply 9 of 9 tested with one 9 of 9 tested with 3 -
passed during 3R2.
(9 snubbers)
failure during 1R2.
failures during 2R4.
4 of 9 tested during 1 of 9 tested and 3R2 through 3R3 with passed during 1R3.
_ no failures.
-
- -
-
-
-
-
-
-
-
.
-
..
.
-
.
.
Unit 2 Snubber Status Reactor Coolant System Drain Line RC-070/071 Piping Evaluation of snubber failures performed to determine if common locations in
-
all three units were experiencing similar failures which could be attributable to system events or whether they were random failures.
Evaluation identified the RC-070/071 piping system as having a high failure
rate. Line RC-070/071 is a 2 inch RCS loop drain to the Reactor Drain Tank.
One of the snubbers on RC-071 had failed (once)in all three units and one of
'
the snubbers on RC-070 had failed (once) in Units 2 and 3.
The failures however, were found to be environmentally related rather than
system transient related. The snubbers which failed in Unit 2 were found to be corrosion related.
Stress analysis performed on the RC-070/071 piping system failed code
allowables for the "As-Found" failure conditions of the U3R3 and U2R4.
These are the only two occasions in the history of Palo Verde snubber testing when code allowables were exceeded.
The stress analysis performed showed that it was the failure of the
RC070HOOA snubber that caused the stresses in the RC070/071 piping system to exceed code allowables. Analysis performed with RC070lIOOA free, passed code allowables with the other two snubbers failed.
Both visual inspections of support steel and liquid penetrant testing of welds
on the RC-070/071 piping system, found no damage or abnormal conditions.
CONCLUSIONS: Failures are environmentally related, internals degraded
over time due to extreme environment. Unit 3 Snubber RC070HOOA replaced last outage. Unit I replaced in 1R2. This snubber should be targeted for snubber reduction analysis.
- -
-
.
.
Unit 2 Snubber Status Small Snubber Failure Mechanisms Nineteen (19) small (1/4 and 1/2 Idp) snubbers failed to pass the acceptance
-
criteria during testing in the Unit 2 outage.
.
With a general population of125 small snubbers, this number (19) represents
-
a 15.2% failure rate.
The failure mechanism fell into 5 difTerent categories:
-
Transient / Vibration (5)
Environment (9)
Installation / Mishandling / Maintenance (2)
Design / Manufacturing (2)
Unknown (1)
The largest number of failures was caused by the environment. Included in the
-
environmental category were snubbers which failed due to:
dried lubrication (5)
corrosion products (4)
Of the live snubber failures which were due to transient events, three of the
-
live may have been involved in a water slug event in the Steam Supply Piping
'
for the Aux Feed Water Terry Turbine.
The only failed small snubber which would cause a stress analysis to exceed
code allowables was on the RC-070/071 lin.
.
.
.
.
.
Unit 2 Snubber Status Large Snubber Failure Mechanisms
.
Nine large snubbers (35 and 100 kip) have failed during the Unit 2 Outage.
- Not all large snubbers have yet been tested in the current outage. 69 of the 71 large snubbers have been tested, the remaining two will be tested when scafTolding for the SG eddy current testingis removed.
Of the 9 failed large snubbers,5 were due to high drag, and 4 were due to
lockup.
Due to the special tooling needed to disassembly Large snubbers, Alllarge,
-
failed snubbers are returned to the manufacturer (PSA) for ERCFA. The PVNGS snubber engineer observed the disassembles.
Disassembly and inspection of known failed snubbers is completed, evaluation
is in progress.
Preliminary findings from the disassembly of the failed 35 and 100 kip
snubbers identified two failure mechanisms, Transient / Vibration and Design / Manufacturing.
All three of the 100 kip failures had signs of brinelling which is indicative of a
vibration induced failure.
The 35 kip failures showed signs of several different failure mechanisms and
one of the 35 kip snubbers had no visualindication of failure mechanism. Two of the 35 kip snubbers exhibited silver flaking on the shaft. One snubber had indication of a system induced failure. The remaining 2 snubbers had indications of excessive vibration.
Vibration data was taken on the same lines as the failed snubbers on Unit 3
while it was in operation. Data has not yet been taken in Unit I but is planned.
The data showed that the vibration of the piping system was within allowable limits of the snubber design,
.-
.. - - -. -
- -
Unit 2 Snubner Status Perspective On Test Criteria Initial testing of snubbers at Palo Verde placed an acceptance criteria for drag
.
of 2% of rated load for all mechanical snubbers. In the 1990-1991 time frame the acceptance criteria was changed to either the actual calculated piping system limit or 5% which ever was lower.
With the new acceptance criteria, any snubber which meet the acceptance
criteria but was greater than 2% were replaced to prevent further degradation.
Palo Verde tests snubbers for both breakaway and drag. This type of testing is
more extensive than many utilities.
To ensure that testing more closely represents the actual operating condition
of the snubber, all snubbers are tested with their applicable extension piece in place. This type of testing places a greater load on the snubber.Many utilities test only the snubber, without the extension in place.
The Palo Verde snubber test machine, manufactured by Enertech, has
consistently been more conservative than the PSA (snubber vendor) machine in that snubbers which we fail on site sometime pass at the vendors factory.
.
.
.
.
.
_ --.
--.
.
.
-
-
.
b
.
>
,
.
.
Unit 3 SBCVInadvertent Opening i
i
,
,
,
.
r
>
I
!
J t
i PVNGS PRESENTATION i
TO NRC REGION V May 24,1993 l
.
ei em-
%
a-
- ye e--
Imu
.. -
.
i
.
Unit 3 SBCVInadvertent Opening Discussion of Event
-
A plant modification to the limit switch actuator arms was
-
completed on SBCV 1005 and a valve quick open was specified
as part of the retest requirements.
l The evolution was treated as a critical evolution and a
-
tailboard was performed. The action to go to Emergency Off was specified if testing did not proceed as expected.
SBCV 1005 was manually isolated and the Master Controller
-
was placed in Manual.
At the SBCV Test Panel, the Mode Selection Switch was moved
-
from OPERATE to VALVE TEST. Because this is a " break
before make" switch and field signals are wired through the contacts of the switch, several of these signals were interrupted long enough to generate actuations in the control circuitry. The following occurred:
.
-
Steam flow and turbine first stage pressure signals to SBCS were interrupted and interpreted by the SBCS as a large load rejection.
-
The large load rejection signal triggered a Quick Open signal.
-
This triggered an open tracking signal w]dch modulates the valves open.
1
'
.
-us
'
Unit 3 SBCVInadvertent Opening Discussion of Event (Cont.)
SBCV's 1001-1006 received 100% modulation demand. The
-
quick open was blocked by the Master Controller being in Manual, but the modulation signal raised the low limit setpoint on the controller causing the valves to open.
Excore linear power approached 104% (CPC compensated i
-
power reached approximsely 109%) and PPS channels received Lo DNBR pre-trips.
SBCS placed in EMERGENCY OFF and:
-
-
All SBCV's close.
-
Reactor power returns to normal.
,
<
i
..}
-
.
,
.
Unit 3 SBCVInadvertent Opening
'
Conclusions From Investigation Plant response to the transient was within analyzed limits.
-
A prior plant modification which added the modulation
-
tracking feature of the valves in order to improve overall performance of the SBCS had an unrecognized effect on the Master Contoller operation.
.
Whenever a Quick Open X or Y is detected by the SBCS, a low limit of 8.33 VDC is applied to the Master Controller. This has the effect of sending a full open modulation demand to SBCV's 1001-1006.
Design review by PVNGS of the the previous modification was
-
inadequate.
The retest requirements from the earlier modification did not
-
specifiy testing with the Mode Selection Switch in VALVE TEST nor measure the output of the Master Contoller in Manual.
l A SBCS Failure Mode and Effects Analysis perfomed in 1991
,
did not address failure modes caused by the Mode Selection
'
Switch.
i Precautions in the CE technical mannual did not adequately
-
address the break before make implications of the Mode Selection Switch and only required placing the Master Controller in manual while turning the switch from OPERATE
,
'
to VALVE TES r n
.
Unit 3 SBCVInadvertent Opening Activities That Address Weaknesses
.
The PVNGS Modification Process is being enhanced and
-
streamlined. Post modification testing is one of the areas in which improvements have been identified. Plant Engineering and Work Controlinterfaces in the retest specifications as well as design engineer accountabilities are being better established.
A verification of post modification testing is currently
-
underway. All U2R4 modifications were listed and those important from a PRA standpoint were selected for verification (11 of 55 selected). Based on the results, a decision i
'
regarding the need for additional verification will be made.
i I
.
.
.
..
.
.
-
.
,
Unit 3 SBCVInadvertent Opening Activities That Address Weaknesw.s
>
(Cont.)
.
Quality Engineering has been monitoring selected
-
modifications each outage that are installed in the Units. QE evaluated installation in accordance with design and adequacy of modification testing. QE is expanding their involvement for future modifications by assigning engineers to provide an independent evaluation of selected modifications from initial design through installation.
Engineering has begun implementation of a program to
-
'
document and trend all aspects of human performance errors.
This program will document those errors that are " caught" during the design process so that trends that indicate
potential poor practices or qualification weaknesses can be
!
identified and acted upon. Even errors caught during independent review processes are intended to be documented.
'
This program differs from the current CRDR process in that it
'
is intended to capture even those errors that are corrected within the Engineering organization before a document goes
"out-the-door".
i
.
.
.
-
-
-
-.
.
.
.
.
.
-
.
Unit 3 SBCVinadvertent Opening
,
i
!
I
^
.
l
-
-
l PVNGS PRESENTATION
TO NRC REGION V May 24,1993
- _
-..
.-
.
Unit 3 SBCVInadvertent Opening Discussion of Event
.
A plant modification to the limit switch actuator arms was
-
completed on SBCV 1005 and a valve quick open was specified
,
as part of the retest requirements.
The evolution was treated as a critical evolution and a
-
tailboard was performed. The action to go to Emergency Off was specified if testing did not proceed as expected.
SBCV 1005 was manually isolated and the Master Controller
-
was placed in Manual.
At the SBCV Test Panel, the Mode Selection Switch was moved
-
-
from OPERATE to VALVE TEST. Because this is a " break before make" switch and field signals are wired through the contacts of the switch, several of these signals were
'
interrupted long enough to generate actuations in the control circuitry. The following occurred:
.
-
Steam flow and turbine first stage pressure signals to SBCS were interrupted and interpreted by the SBCS as a large load rejection.
-
The large load rejection signal triggered a Quick Open signal.
This triggered an open tracking signal which modulates
-
the valves open.
Unit 3 SBCVInadvertent Opening Discussion of Event (Cont.)
SBCV's 1001-1006 received 100% modulation demand. The
quick open was blocked by the Master Controller being in Manual, but the modulation signal raised the low limit setpoint on the controller causing the valves to open.
Excore linear power approached 104% (CPC compensated
-
power reached approximately 109%) and PPS channels received Lo DNBR pre-trips.
i
,
SBCS placed in EMERGENCY OFF and:
-
All SBCV's close.
-
Reactor power returns to normal.
,
, -,
.
Unit 3 SBCVInadvertent Opening Conclusions From Investigation Plant response to the transient was within analyzed limits.
-
A prior plant modification which added the modulation
-
tracking feature of the valves in order to improve overall
,
performance of the SBCS had an unrecognized effect on the Master Contoller operation.
Whenever a Qtdek Open X or Y is detected by the SBCS, a low limit of 8.33 VDC is applied to the Master Controller. This has
,
the effect of sending a full open modulation demand to SBCV's 1001-1006.
Design review by PVNGS of the the previous modification was
-
inadequate.
,
The retest requirements from the earlier modification did not
-
specifiy testing with the Mode Selection Switch in VALVE TEST nor measure the output of the Master Contoller in Manual.
A SBCS Failure Mode and Effects Analysis perfomed in 1991
-
did not address failure modes caused by the Mode Selection Switch.
Precautions in the CE technical manaual did not adequately
-
address the break before make implications of the Mode Selection Switch and only required placing the Master Controller in manual while turning the switch from OPERATE to VALVE TEST.
. - -
-
-
- - - -.
. - - - - - -
_ ------ _----_
- - - - _ - - - - - - - - - _ - - - - - - - - _ - _ - - - - -
'
'
Unit 3 SBCVInadvertent Opening Activities That Address Weaknesses
.
The PVNGS Modification Process is being enhanced and
-
streamlined. Post modification testing is one of the areas in which improvements have been identified. Plant Engineering and Work Control interfaces in the retest specifications as well as design engineer accountabilities are being better established.
A verification of post modification testing is currently
underway. All U2R4 modifications were listed and those important from a PRA standpoint were selected for verification (11 of 55 selected). Based on the results, a decision regarding the need for additional verification will be made.
-
- _ _ _ _ _ - _ - - _ _ _.
- - _
- - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _
.
'
Unit 3 SBCVinadvertent Opening
<
Activities That Address Weaknesses (Cont.)
_
Quality Engineering has been monitoring selected
modifications each outage that are installed in the Units. QE evaluated installation in accordance with design and adequacy of modification testing. QE is expanding their involvement for future modifications by assigning engineers to provide an independent evaluation of selected modifications from initial design through installation.
Engineering has begun implementation of a program to
-
document and trend all aspects of human performance errors.
This program will document those errors that are " caught" during the design process so that trends that indicate potential poor practices or qualification weaknesses can be identified and acted upon. Even errors caught during independent review processes are intended to be documented.
This program differs from the current CRDR process in that it is intended to capture even those errors that are corrected within the Engineering organization before a document goes
"ou t-the-door".
-
.
Reactor Trip Breaker Status
,
PVNGS PRESENTATION TO NRC REGION V May 24,1993
.
-
-
.
_.
..
Reactor Trip Breaker Status
-
Contents
.
I.
Overview II.
Summary of 1992 RTB Root Cause of Failures III.1993 RTB Performance IV. Corrective Actions For GE AKR-30 Breakers V.
Relationship With GE VI. Decision To Replace AIUt-30's With DS-41G's VII. Perspective - Closing Remarks
_
Reactor Trip Breaker Status Overview
.
On March 31,1992 a Westinghouse DS-20G reactor trip
-
breaker failed to fully open during a surveillance test. In the previous two weeks a General Electric AKR-30 reactor trip breaker had failed to close on some occasions during similar testing.
With the exception of the original Westinghouse breaker
-
failure to fully open, no breaker tested in an as-found condition has failed to open when required.
The Westinghouse DS-206 reactor trip breakers have only
-
experienced one testing anomaly since the March 1992 failure. This was a cell switch failure which affected operation in the TEST position only. No abnormal performance of UV devices has occurred.
The originalproblem with the General Electric AKR-30
-
reactor trip breaker was a random trip when closed. This problem has been minimized and has not recurred since all breakers have had the corrective action implemented (UV armature to trip paddle gap adjustment).
All 1993 problems with the Gencral Electric AIU1-30 breaker
-
are related to the UV device.
.
.
Reactor Trip Breaker Status Stunmary of 1992 RTB Root Cause of Failures Westintrhouse With the exception of the March 31,1992 failure to fully epen
-
event, the Westinghouse DS-206 RTB's performed without incident during 1992.
Since the condition wldch caused the Westinghouse DS-206
-
breaker to hang up in the intermediate position did not repeat itself during testing, a definitive root cause could not be determined.
The apparent cause of failure was a combination of an
-
inadequate maintenance program and the marginal opening force inherent in the DS-206 design.
Corrective Actions:
-
-
Optimize the Maintenance Program (April 1992)
-
Improve the Training and Qualification Program (October 1992)
-
Incorporate Westinghouse Tecimical Bulletin Into PM Procedure (June 1992)
-
Evaluate, Design, and Implement Design Change to replace DS-206 breakers with model DS-416. (1R4,3R4, 2R5)
.. -
.
.
Reactor Trip Breaker Status Sununary of 1992 RTB Root Cause of Failures i
General Electrie The problems experienced during surveillance testing in
-
1992 can be categorized as follows:
-
Breaker goes "CLOSE-TRIP" on some closure attempts.
This was original problem investigated beginning in April 1992.
-
UV device armature found in intermediate position when deenergized.
\\
Problem, as reported, never manifested itself during troubleshooting. Root cause team attributed probable
'
cause to foreign materialin UV picup adjustment spring.
-
UV device armature-rivet interference.
Isolated case where teclulician verified rivet clearance incorrectly.
-
UV device fails to reset when the coil is " hot".
Caused by limited design margins of the UV device.
Does not impact ability of UV device to perform its safety function.
CE Info-Bulletin issued in 1984 and GE letter issued in 198G describe illis problem, with the GE letter stating that coil resistance could increase by 25% when the coil is ho.
.
Reactor Trip Breaker Status Summary of 1992 Root Cause of Failures (Cont.)
.
PVNGS surveillance procedures incorporated guidance from these-documents.
In October, after extensive testing at the GE facility in
Chamblee, GA, GE issued the cause of the CLOSE-TRIP t
behavior which agreed with the original root cause of failure determination made by the PVNGS team in April.
The cause was attributed to insufficient gap between the undervoltage device trip paddle adjustment screw and trip paddle clamp. Breakers with newer mechanisms appear to not be susceptible to tlds. Buffer adjustments may contribute to the shock that causes the trip paddle to rebound.
Setting the clearance to the currently recommended
-
clearance will minimize CLOSE-TRIPS but will not eliminate them.
l
l
-
.
.
Reactor Trip Breaker Status 1993 RTB Performance Westinchouse Failure of CELL Switch in TESTPosition n
-
In March, a Westinghouse breaker failed ~ to open with the breaker in the TEST position.
-
Cause attributed to deformation of the spring return mechanism of the CELL switch.
-
Does not affect the ability of the breaker to open in
,
CONNECT position.
Two failures of these switches have occurred at Palo
-
Verde. Tids type of failure was identified by Westinghouse in 1987 and also in NRC IN 87-61.
General Electric Failure of UVDevice Armature to Reset When UV Coilis Hot
=
This problem is discussed under 1992 RTB Root Cause of Failure.
.
.
Reactor Trip Breaker Status 1993 RTB Performance General Electric (Cont.)
,
UVDevice Armature Contact With Frame or Mounting Bolt
=
Head Resulting in Inconsistent Pickup Voltage Measuremen ts
-
One UV device failed acceptance criteria as a result of the armature rubbing against the edge of the front of the frame. A second failed the pickup criteria due to the armature head rubbing against the head of one of the mounting bolts (discussed below). A tinrd (new from the warehouse) failed inspection due to the armature head rubbing against the edge of the top of the frame.
-
UV devices procured prior to 1986 had the potential for the mounting screw bolt heads to interfere with the movement of the armature. This problem was associated with certain devices that may have been assembled with the armature riding higer in the frame.
GE issued a Service Advisory Letter (SAL) in 1985 alerting customers of this problem. Palo Verde inspected their UV devices and took the action required at that time but did not recognize that the problem could potentially occur anytime the UV device was dissasembled and reassembled.
Palo Verde had the older style UV devices on 2 of their 6 installed breakers and on 1 spare breaker.
Although one of the older syle UV devices failed UV coil acceptance testing during planned maintenance in the electric shop attributed to rubbing of the armature head aginst the bolt heads, none of the devices failed to trip the breaker when required.
.
-.
.
. _ _ _ _ _ _.
_
i
.
.
i Reactor Trip Breaker Status 1993 RTB Performance General Electric (Cont.)
Failure of UVDevices to Meet Armature Pickup or Dropout Acceptance Criteria During Planned Maintenance j
-
Major revision to PM procedure completed October 26, J
1992 incorporating all available vendor information.
-
UV device " dropout voltage" acceptance criteria changed from 37.5-75 vde to 38-43 vdc.
,
-
Between November 1992 and January 1993 two UV devices failed the new acceptance criteria during planned maintenance.
i
-
In February, both Unit 2 and Unit 3 experienced difficulty with UV devices meeting acceptance criteria, including new UV devices withdrawn from the warehouse.
Working with GE, determined that the dropout voltage
-
was to be measured with the coil" cold" vs. " hot" and was 7erly restrictive for field acceptance.
-
CE had directed PVNGS, during Startup, to take the original criteria while the coil was hot. The latest GE technical manual did not specify under what conditions to take the data. GE had reviewed the PM procedure during 1992 and did not identify this as a concern.
.
. _. _ _,.
.
~.
.
.
Reactor Trip Breaker Status Corrective Actions For GE AKR-30 Breakers
Palo Verde efforts have looked at all programs that support
the reactor trip breakers. These include: Training and -
Qualification, Procedures, Parts Procurement, Vendor Interface, Root Cause Determinations, Engineering Involvement and Design.
Three detailed root cause of failure efforts followed by
apparent cause root cause determinations on each reator trip breaker anomaly were completed.
Below is a partial list of corrective actions implemented:
-
Obtained revised GE Technical Manual and incorporated the requirements into the maintenance procedures. Many additional changes made to procedures to incorporate specific lessons learned and add clarity.
-
Received GE concurrence on armature to-trip-paddle gap measurement of.030 in being necessary to minimize CLOSE-TRIPS.
-
Reestabished two contractural relationships with GE (1 nuclear,1 non-nuclear) for technical manual changes
,
including SAL's.
'
-
Vendor documents now get front end review for significance and applicability. CRDR's are issued as required.
-
Developed separate training program for reactor trip breakers.
-
Implemented a specific qualification for electricians performing reactor trip breaker maintenance.
.
l
.
.
Reactor Trip Breaker Status Corrective Actions For GE AKR-30 Breakers (Cont.)
-
Implemented an enhanced QC monitoring program for breaker maintenance.
Revised purchasing documents to ensure GE will
-
provide UV devices properly set up.
-
Implemented the use ofimproved DC power supplies for breaker maintenance.
-
Prohibited the former practice of replacing the coils on UV devices. Now must replace the entire UV device assembly.
-
Established a Root Cause of Failure Program Manager.
The program was recently recognized by INPO as a Good Practice.
Enhanced the quarantine procedure and the role of the
-
STA in the quarantine process.
-
Worked with GE to evaluate short vs. Iong trip paddles, improved design for trip paddle, and acceptance criteria for UV device testing.
Palo Verde root cause determinations have been thorough
and technically sound. Final aggreement by GE on PVNGS recommendations substantiate this.
Implementation, as intended, of recognized improvements
-
takes time and enormous effort to reach all affected organizations and personne I-
.
Reactor. Trip Breaker Status Corrective Actions For GE AKR-30 Breakers (Cont.)
>
.
Current behavior of the GE AKR-30' breakers is now well
-
understood and programs are close to being optimized.
Despite all the efforts, the GE RTB's (particularly the UV devices) will continue to exhibit certain characteristics.
These are due to inherent design limitations which can be minimized by the corrective actions implemented.
What to Expect UV device not resetting after prolonged energization.
-
Rejection of new UV devices due to poor workmanship,
-
material defects, open UV coils.
Rubbing of some UV device armatures against frame.
-
,
Possible CLOSE-TRIPS due to UV trip paddle and trip shaft
-
cInmp interactio _.
.
Reactor Trip Breaker Status Relationship With GE 1992 Areas Identified For Improvement
-
Lack of knowledge about GE organizations.
-
Quality of engineering evaluations.
-
Quality of repair facility refurbish 2nents.
UV device dedication by San Jose Nuclear group.
-
-
Organizational responsiveness to utility emergent issues.
Efforts Undertaken to Improve
-
-
GE Senior Management meeting with VP and Director Engineering.
-
Conducted two GE Style Workout sessions withjoint GE and Palo Verde participation - one in Phoenix and one in San Jose.
-
Worked jointly with GE during testing in Chamblee.
-
Vendor Quality assessments of the Chamblee repair facility and San Jose procurement dedication.
,
.
.
.
Reactor Trip Breaker Status Relationship With GE (Cont.)
.
Changes Made
-
Established an E-Maillink with GE
-
Established a single point of contact within PVNGS and
-
GE for communicating technicalissues/ requests.
Pulled GE repair facility from qualified vendor list,
-
worked with them to upgrade program for breakers,
'
and re-approved their program for Palo Verde work.
-
Influenced the transfer of UV dedication from San Jose to Chamblee, GA
,
.
.
Reactor Trip Breaker Status GE AKR-30 Breaker Upgrade Factors Influencing Decision To Replace
-
Maintenance of the UV devices required continuous
-
attention
-
Lack of timely support from GE on design concerns
-
Quality concerns with GE refurbislunents and parts replacement
-
Costs of GE refurbishment Options Considered
-
-
Replace UV device with a second shunt trip device
,
Replace / Modify the existing UV device trip shaft paddle
-
Modify the UV device
-
-
Replace the GE AKR-30 breaker with Westinghouse DS-41G
-
Replace the GE AKR-30 breaker with alternate manufacturer Why Westinghouse DS-41G?
-
-
PVNGS Maintenance and Engineering familiarity
-
Proven reliability in Class 1E RTSG applications Upgrade path for existing Westinghouse DS-20G RTB's
-
-
Existing switchgear compatibility
,
,
Reactor Trip Breaker Status
_
l Perspective - Closing Remarks l
l Most of the PVNGS activities which affect reactor trip l
breakers have had improvements made.
Sensitivity to reactor trip breaker performance problems is
,
now meeting expectations. At times, there have been
'
examples of being overly sensitive.
While several of the problems experienced have represented
the potential for a breaker to fail to perform its required safety function, only the Westinghouse RTB, an'one occasion, has failed to open in an as found condition.
With all the investigations and improvements completed,
many of the same breaker performance problems that occurred a year ago on the GE RTB can still occur. This can be attributed to the following:
-
The GE breaker has design limitations. Even brand new breakers could experience the CLOSE-TRIP problem with breaker mechanisms parts within design tolerances.
The UV device has proven to be reliable but it is not a
-
precison piece of equipment. Also,it can be easily damaged if mishandled. The positive trip verification requirement has included allowable margin to ensure the device will trip the breaker despite its limitations.
However, rejection of devices because of failure to meet acceptance criteria can be expected to continue.
-
The ability of the armature on the UV device to pickup when the coilis " hot"is marginal. This feature does not impact the ability of the device to open the breaker when require.
.
.
.
,
Reactor Trip Breaker Status Corrective Actions For GE AKR-30 Breakers Palo Verde efforts have looked at all programs that support
the reactor trip breakers. These include: Training and Qualification, Procedures, Parts Procurement, Vendor Interface, Root Cause Determinations, Engineering Involvement and Design.
Three detailed root cause of failure efforts followed by
apparent cause root cause determinations on each reator trip breaker anomaly were completed.
Below is a partiallist of corrective actions implemented:
-
Obtained revised GE Technical Manual and incorporated the requirements into the maintenance procedures. Many additional changes made to procedures to incorporate specific lessons learned and add clarity.
-
Received GE concurrence on armature-to-trip-paddle gap measurement of.030 in. being necessary to minimize CLOSE-TRIPS.
-
Reestabished two contractural relationships with GE (1 nuclear,1 non-nuclear) for tecimical manual changes including SAL's.
-
Vendor documents now get front end review for significance and applicability. CRDR's are issued as
.
required.
-
Developed separate training program for reactor t ip r
breakers.
Implemented a specific qualification for electricians j
-
performing reactor trip breaker maintenanc.
.
.
Reactor Trip Breaker Status Corrective Actions For GE AKR-30 Breakers (Cont.)
Implemented an enhanced QC monitoring program for
-
breaker maintenance.
Revised purchasing documents to ensure GE will
-
provide UV devices properly set up.
Implemented the use of improved DC power supplies for
-
breaker maintenance.
Prohibited the former practice of replacing the coils on
-
UV devices. Now must replace the entire UV device
,
assembly.
Established a Root Cause of Failure Program Manager.
-
The program was recently recognized by INPO as a Good Practice.
Enhanced the quarantine procedure and the role of the
-
STA in the quarantine process.
Worked with GE to evaluate short vs. long trip paddles,
-
improved design for trip paddle, and acceptance criteria for UV device testing.
Palo Verde root cause determinations have been thorough
-
and technically sound. Final aggreement by GE on PVNGS recommendations substantiate this.
Implementation, as intended, of recognized improvements j
takes time and enormous effort to reach all affected
'
organizations and personne.
-.
.
.
Reactor. Trip Breaker Status Corrective Actions For GE AIM-30 Breakers (Cont.)
.
Current behavior of the GE Alm-30' breakers is now well
-
understood and programs are close to being optimized.
Despite all the efforts, the GE RTB's (particularly the UV devices) will continue to exldbit certain characteristics.
These are due to inherent design limitations which can be minimized by the corrective actions implemented.
Mat to Expect UV device not resetting after prolonged energization.
-
Rejection of new UV devices due to poor workmanship,
-
material defects, open UV coils.
Rubbing of some UV device armatures against frame.
-
Possible CLOSE-TRIPS due to UV trip paddle and trip shaft
-
clamp interactio *
.
Reactor Trip Breaker Status Relationship With GE (Cont.)
_
Changes Made
-
-
Established an E-Maillink with GE Established a single point of contact within PVNGS and
-
GE for communicating technicalissues/ requests.
-
Pulled GE repair facility from qualified vendor list, worked with them to upgrade program for breakers, and re approved their program for Palo Verde work.
Influenced the transfer of UV dedication from San Jose
-
to Chamblee, GA
.-
Reactor Trip Breaker Status Relationship With GE 1992 Areas Identified For Improvement
-
Lack of knowledge about GE organizations.
-
-
Quality of engineering evaluations.
'
-
Quality of repair facility refurbislunents.
UV device dedication by San Jose Nuclear group.
-
-
Organizational responsiveness to utility emergent issues.
Efforts Undertaken to Improve
-
-
GE Senior Management meeting with VP and Director Engineering.
l Conducted two GE Style Workout sessions withjoint GE
-
and Palo Verde participation - one in Phoenix and one in San Jose.
'
-
Worked jointly with GE during testing in Chamblee.
-
Vendor Quality assessments of the Chamblee repair facility and San Jose procurement dedicatio _
.
.-
..
.
_
.
.-
...
..-
_
_
,
.
.
'!
h Reactor Trip' Breaker Status l
Relationship With GE'
(Cont.) -
t f
.
.
Changes Made
-
t
-
Established an E-Maillink with GE
Established a single point of contact witidn PVNGS and'-
'
-
GE for communicating technicalissues/ requests.
-
Pulled GE repair facility from qualified vendor list, worked with them to upgrade program ~ for breakers, r
and re-approved their program for Palo Verde work.
-
Influenced the transfer of UV dedication from San Jose to Chamblee, GA
!
i
-
.
-
...
-..
.
-
-
-
-
.
,-
'
..
.
.
Reactor Trip Breaker Status GE AKR-30 Breaker Upgrade Factors Influencing Decision To Replace
-
Maintenance of the UV devices required continuous
-
attention
-
Lack of timely support from GE on design concerns
-
Quality concerns with GE refurbishments and parts replacement
-
Costs of GE refurbishment Options Considered
-
Replace UV device with a second shunt trip device
-
-
Replace / Modify the existing UV device trip shaft paddle
,
-
Modify the UV device
-
Replace the GE AKR-30 breaker with Westinghouse DS-416
-
Replace the GE AKR-30 breaker with alternate manufacturer Why Westinghouse DS-41G?
-
PVNGS Maintenance and Engineering familiarity
-
Proven reliability in Class IE RTSG applications
-
Upgrade path for existing Westinghouse DS-20G RTB's
-
,
-
Existing switchgear compatibility j
,
-
.
,
,
Reactor Trip Breaker Status Perspective - Closing Remarks Most of the PVNGS activities which affect reactor trip
-
breakers have had improvements made.
Sensitivity to reactor trip breaker performance problems is
-
now meeting expectations. At times, there have been examples of being overly sensitive.
While several of the problems experienced have represented
-
the potential for a breaker to fail to perform its required safety function, only the Westinghouse RTB, on one occasion, has failed to open in an as-found condition.
With all the investigations and improvements completed,
-
many of the same breaker performance problems that occurred a year ago on the GE RTB can still occur. This can be attributed to the following:
-
The GE breaker has design limitations. Even brand new breakers could experience the CLOSE-TRIP problem with breaker mechanisms parts within design tolerances.
The UV device has proven to be reliable but it is not a
-
precison piece of equipment. Also,it can be easily damaged if mishandled. The positive trip verification requirement has included allowable margin to ensure the device will trip the breaker despite its limitations.
However, rejection of devices because of failure to meet acceptance criteria can be expected to continue.
-
The ability of the armature on the UV device to pickup when the coilis " hot"is marginal. This feature does not impact the ability of the device to open the breaker when required.