IR 05000528/1990042
| ML17305B311 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 12/26/1990 |
| From: | Huey F NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML17305B309 | List: |
| References | |
| 50-528-90-42, 50-529-90-42, 50-530-90-42, NUDOCS 9101150080 | |
| Download: ML17305B311 (77) | |
Text
0
,U.S.
NUCLEAR REGULATORY COMMISSION
REGION V
Report Nos.
50-528/90-42, 50-529/90-42 and 50-530/90-42 Docket Nos.
50-528, 50-529, 50-530 License Nos.
NPF-41, NPF-51, NPF-74 Licensee:
Arizona Public Service Company P.
0.
Box 53999, Station 9012 Phoenix, AZ. 85072-3999 Facility Name:
Palo Verde Nuclear Generating Station Units 1, 28
Inspection Conducted:
October 1 through November 9, 1990 Inspectors:
C.
M. Caldwell, Team Leader, RV A.
S.
Gautam, Assistant Team Leader, NRR D.
S. Acker, Reactor Inspector, RV F.
S.
Gee, Reactor Inspector, RV A.
L. Della Grecca, Reactor Inspector, RI H. Sinah, Electrical Engineer, Atomic Energy Canada Ltd (AECL)
M. Goel, Mechanical Engineer, AECL J.
T. Hailer, Electrical Engineer, AECL Approved By:
uey, ie Engineering Section (~i~a/q o a e igne Inspection Summary:
Ins ection on October 1 throu h November
1990 (Re ort Numbers an Areas Inspected:
Electrical Distribution System Functional Team Inspection in accordance with the provisions of Temporary Instruction (TI) 2515/107 to the NRC Inspection Manual.
Results:
Of the areas inspected, three violations were identified.
The viVo a Tons pertained to 10 CFR Appendix B design control requirements (Paragraph 2.2), procedures for revising calculations (Paragraph 2.1),
and maintenance (Paragraph 4. 3) at Unit(s) 1, 2 and 3.
In addition, the team identified one non-cited violation concerning cable trays (Paragraph 4. 1).
910ii50080 90i227 PDR ADQCK 05000528
DETAILS Persons Contacted:
Arizona Nuclear Power Pro ect (ANPP)
- R. Adney, Plant Manager, Unit 3
~J. Bailey, Vice President, Nuclear Safety 8 Licensing B. Ballard, Director, Quality Assurance.
T. Bradish, Manaqer, Compliance
- N. Bryant, Technlca1 Assistant Manager, Training W.
Conway, Executive Vice President, Nuclear
- E. Dotson, Director, Engineering 8 Construction R. Flood, Plant Manager, Unit 2
- J. Flomerfelt, Manager, Nuclear Records Management
~S. Guthrie, Deputy Director, QA/QC
- L, Henson, Electrical Supervisor, Site Nuclear Engineering M.
Hodge Manager, Mechanical/Chemical
~W. Ide, Plant Hanager, Unit 1
- S. Karimi, Site Nuclear Engineering
- S. Kesler, Nuclear Engineering
- J.'evine, Vice President, Nuclear Power Production
~W. Marsh, Director, Plant Operations 8 Maintenance G. Oyerbeck, Director, Technical Support
- S. Pierce, Site Nuclear Engineering
- M. Powell, Manager, Nuclear Licensing F. Prawlocki, Manager, Engineering Assurance
~W. Quinn, Director, Nuclear Licensing
"M. Radoccia, Manager, Site Nuclear Engineering Department
- R. Rouse, Supervisor, Complian'ce
~T. Shriver, Assistant Plant Manager, Unit 2 E.
Simpson, Vice-President, Engineering 8 Construction
- G. Sowers, Technical Assistant to Director, Site Technical
- J.
Summy, Manager, System Engineering and Technical Support W. Webster, Manager, Component and Specialty Engineering R. Younger, Manager, Maintenance Standards Other NRC Personnel
"K.
- R.
+W.
- J Perkins, Deputy Division Director, DRSP, RV Huey, Engineering Section Chief, RV Ang, ProJect Inspector, RV Sloan, Resident Inspector, RV Other Personnel AJ
- R 5kJ
- R Panesar, Mechanical Section Head, AECL Basu, Engineer, PG&E Gura, Engineer, ERCE Draper, Site Representative, SCE Epstein, Consultant, Devonrue Henry, Salt River Project, Site Representative
e
The inspectors also interviewed other licensee and contractor personnel during the course of the inspection.
- The asterisk denotes those who attended the Exit meeting held with NRC inspection team on November 9, 1990.
'
SUMMARY A Nuclear Regulatory Commission (NRC) team conducted an electrical distribution system functional inspection (EDSFI) at the Palo Verde Nuclear Generating Station (PVNGS).
The inspection was conducted by personnel from Region V, Region I, the Office of Nuclear Reactor Regulation (NRR), and contractor personnel from Atomic Energy Canada Ltd (AECL) from October
through November 9, 1990.
The NRC inspection team reviewed the design and implementation of the plant electrical distribution system (EDS) and the adequacy of associated engineering and technical support.
To accomplish this, the team reviewed the design of the electrical and mechanical systems and equipment affecting the EDS, examined installed EDS equipment, reviewed test programs, reviewed procedures affecting the EDS, and determined the adequacy and interfaces of technical disciplines and functions by interviewing appropriate corporate and site personnel.
The team found that the engineering organizations were providing adequate engineering and technical support to the site.
There were a number of strengths noted by the team during the inspection.
For example:
Design documents were easy to read and readily available.
Training of system engineers appeared to be good and they were knowledgeable of their systems.
Engineering personnel were knowledgeable of plant design and site personnel
$ e. g., maintenance and operations personnel)
were knowledgeable of plant equipment.
It appeared that site nuclear engineering was helping to overcome some of the engineering weaknesses experienced in the past.
In most cases, engineering calculations had conservative assumptions and conclusions were technically sound.
In general, the team considered the design implementation of the EDS at Palo Verde to be acceptable.
However, the team found that APS had not evaluated the EDS to demonstrate that it would satisfy the requirements of GDC-17 (for electric power systems)
specified in Appendix A to 10 CFR 50, under all allowed plant operatinq conditions.
For example, the effects of the addition of non-Class 1E loads (40 MVA) being automatically transferred from the unit auxiliary transformer to the startup transformer on a fast bus transfer and the effects of this additional load on the evaluated voltages to safety related components were not considered.
Subsequent APS calculations, performed in response to the team findings, confirmed that the EDS design will satisfy the requirements of GDC 17 with the following exception.
The offsite power system was not designed to minimize the likelihood of simultaneous failure of both sources when the three units are powered from two startup transformers and a fast bus transfer of non-Class lE loads from the
unit auxiliary transformer to the startup transformer occurs in one unit with another unit in startup.
In response to the team, APS identified five occasions in which the rating for the secondary winding of at least one startup transformer would have been exceeded if a reactor trip, turbine trip, or loss of coolant accident had occurred in one of the units.
On one of these occasions, excess current may have occurred on the secondary of two startup transformers that were supplying power to Unit 2 if one of the above scenarios had taken place.
This may have resulted in a loss of both offsite power sources to Unit 2.
As a result, administrative controls for the loading of the startup transformers were implemented by the licensee.
An engineering problem area noted during the inspection concerned calculation errors which resulted in several instances of actual bus loadings being higher
.than originally considered.
The team considered that these findings were examples of deficiencies in preparation, control, and maintenance of the design basis for the EDS.
l, One error of particular note concerned changes to an electrical calculation that were not appropriately incorporated by APS into other related loading calculations.
This could have resulted in confusion as to what the proper load values were for the emergency diesel generators and which values should be used during the performance of design work.
A second example involved the effect of a water hammer on the spray pond system.
In particular, one train of the system had experienced a number of water hammers and APS implemented a
1985 Bechtel modification to eliminate the possibility of further water hammer.
However, a walkdown of the spray pond modification by the team found that it was not installed as designed and probably would not be able to perform its intended function in that a vent line, installed to prevent water hammer, was installed under water instead of above water as a result of erroneous design drawings.
The inspection team recognized that the licensee has made a number of enhancements to the design control process since implementation of this modification.
However, there were no modifications implemented under the new process that could be evaluated by the team to fully assess these program changes.
Although APS was in the process of implementing changes to the engineering organization structure and undertaking an effort to reestablish the design basis of the plant, the team was concerned that the licensee had performed several recent assessments of the EDS design adequacy, yet did not find the types of calculation problems noted by the team.
Mith regard to the design basis reconstitution effort, APS indicated that, as a result of concerns noted during the EDSFI, they felt the need to perform an accelerated review of EDS calculations prior to conducting the design basis effort on the EDS.
APS currently intends to perform a calculation reverification program that will perform a significance overview, a
prioritization, and a calculation review that will be performed on a timeline that is commensurate with the importance of the specific calculation.
Included in this effort will be a retrieval of assumptions for specific calculations.
APS indicated that, through this effort, approximately
electrical calculations, approximately 100 mechanical calculations, approximately 31 I8C calculations, and approximately 32 civil calculations would be reviewed within the next two years.
The remaining calculations
0 (e. q., nonprotective system - balance of plant devices)
are intended to be reviewed within the next five years.
This effort will then be factored into the design basis reconstitution effort as it is implemented for the respective systems.
The team believed that the scope of the site testing and maintenance program for EDS equipment was generally adequate.
However, there were a number of safety related components that were not being tested to demonstrate that they would function when called upon.
Several examples were noted (e.g.,
some molded case control power breakers were not being tested and there was no maintenance requirement implemented for some 120 VAC distribution panels).
The licensee indicated that they planned to test them in the future.
The team considered that the lack of surveillances or preventive maintenance
'n various safety related equipment indicated a weakness in preventive maintenance and testing programs.
Although APS has started to implement a
preventive maintenance task force (PMTF) to address weaknesses in this area, the team noted that some of the items reviewed were not being considered for inclusion in any preventive maintenance (PM) task or surveillance as of this inspection.
Thus, the team considered that the licensee may want to continue to challenge the assumptions made in the PM effort.
The inspection team considered that engineering and technical support to the plant were getting better (in relation to the past).
However, several weaknesses were identified.
For example:
There were inadequate instructions defining the responsibilities of the component engineers or their interface with system engineers.
There were inadequate instrucbons to ensure that load data on single line diagrams (nameplate ratings) were used in calculations instead of the actual loading data.
In response, the licensee issued instructions to define responsibilities of the component engineers, and to specify the correct information to be used in loading data calculation io INTRODUCTION Before this inspection the Nuclear Regulatory Commission (NRC) staff had identified several electrical distribution system (EDS) deficiencies during electrical inspections at vario'us operating plants in the country.
The Special Inspection Branch initiated inspections of the EDS at operating nuclear plants after it determined that such deficiencies could affect the EDS power sources and equipment and could compromise the design safety margins of nuclear plants.
Examples of these deficiencies included unmonitored and uncontrolled load growth on safety buses, inadequate engineerinq modifications, design calculations, testing of EDS equipment, and qualification of commercial-grade equipment used in safety-related applications.
The NRC considered one cause of these deficiencies to be inadequate engi neeri ng and technical support.
As a result, Temporary Instruction (TI) 2515/107, to the NRC Manual entitled,
"Electrical Distribution System Functional Inspection (EDSFI)," was issued.
The objectives of this inspection were to assess the performance capabi lity of the Palo Verde EDS and the capability and performance of the licensee's engineering and technical support in this area.
For the purpose of this inspection, the EDS included all emergency sources of power and associated equipment providing quality power to systems relied on to remain functional during and following design basis events.
The EDS components included the emergency diesel generators 125 Vdc Class lE batteries; two offsite circuits from the 525 kV offsite power grid switchyard; distribution transformers; 4160 V switchgear; 480 Vac load and motor control centers; 120 Vac, and 125 Vdc power panels; and battery chargers, inverters, associated buses, breakers, relays and devices.
The team reviewed the adequacy'f emergency onsite and offsite power sources for EDS equipment, the 'regulat)on of power to essential loads, protection for postulated fault currents, and coordination of the current interrupting capability of protective devices.
The team reviewed mechanical support systems affecting the EDS, including air start, lube oil, and cooling systems for the emergency d>esel generator as well as cooling and heating systems for EDS equipment.
The team physically examined originally installed EDS equipment and modifications for configuration and ratings and,reviewed qualification testing and calibration records.,
The team also assessed the capability and performance of the licensee's engineering/technical support functions with regard to organization and key staff, timely and adequate root-cause analysis for failures and recurring problems, and engineering involvement in design and operations.
The team reviewed EDS conformance with General Design Criteria (GDC) 17 and 18 and appropriate criteria of Appendix B to 10 CFR Part 50.
The team reviewed plant Technical Specifications (TS), the Updated Final Safety Analysis Report (UFSAR), and appropriate safety evaluation reports to verify that technical requirements and licensee commitments were being met.
The areas reviewed and the safety significance of identified deficiencies are described in Sections 2, 3, 4, and 5 of this report.
Conclusions are
I
2.0 provided at the end of each of these sections.
A summary of the conclusions and weaknesses is given in Section 6 of this report.
ELECTRICAL SYSTEMS The team reviewed a sample of specific electrical design attributes at each AC and DC level of the EDS.
This included verifying the adequacy of plant load calculations for the regulation of electrical loads needed for the safe shutdown of the plant, overcurrent protection calculations for short circuit and ground faults, and the sizing and coordination of protective devices.
The team reviewed appropriate design documents related to electrical loads associateg with the EDS.
The documents reviewed addressed design calculations for AC and DC system loading, voltage regulation during normal and degraded conditions, voltage regulation during sequencing of safety-related loads onto the preferred power supply and the emergency diesel generators (EDGs),
degraded voltage relay setpoints, Class 1E battery selection, short circuit and ground-fault analysis, system and equipment overcurrent protection, protective device coordination, and the protection of the EDS from power surges.
Particular attention was given to a selected sample load path or "vertical slice" through the Class 1E electrical distribution system.
The team also reviewed procedures and guidelines governing EDS design calculations, design control, and plant modifications; reports on EDG qualification tests, and AC system voltages during degraded voltage conditions; and EDS single-line, schematic, and protective relay setting drawings.
The 4160 V (4. 16 kV) Class lE distribution system, for each of the three units at the nuclear generating station, consisted of the power feeds from the preferred or offsite grid source, two redundant switchgear buses, standby/onsite EDGs, and the power distribution equipment and circuits to accident mitigating loads.
Each unit had two independent EDGs to provide standby power.
One EDG was provided for each of the safety divisions, with each EDG unit capable of being connected to supply its corresponding 4160 V safety bus.
Each of the two redundant Class 1E 4160 V buses for each unit was normally energized from the preferred offsite power source (525 kV grid) through one of three startup transformers (525 to 13.8 kV) and an engineered safety features (ESF)
transformer (13.'8 to 4.16 kV).
The Class lE 4160 V buses primarily supplied accident mitigating pump motor loads and had three safety-related 480 Y load centers per bus.
All redundant safety-related loads were divided between the redundant 'Class lE buses.
Since the 4160 V Class lE buses were normally served from the preferred power source, no automatic transfer of these buses, other than to the EDGs upon loss of offsite power, was required.
Following unit startup, large nonsafety-related loads, including the reactor coolant pump motors were manually transferred from the startup transformers to a unit auxiliary transformer connected to the respective unit's generator.
However, upon a plant emergency trip, the non-safety loads were automatically transferred back to the startup transformers.
To assess the adequacy of the regulation of EDS loads, the team reviewed the characteristics of (1) the offsite electrical grid to verify
0
P reliability of power; (2) preferred source transformers (i.e. startup and ESF) to assess their ratings and connections to the safety buses; and (3)
EDGs to assess the adequacy of their load capacity and their ability to start and accelerate the assigned safety loads in the required time sequence.
The team also reviewed (j.) the transient, steady-state, maximum and minimum voltages, and transient frequencies associated with the 4160 V system; (2) the capability of the 4160V system to sustain a
single failure concurrent with a loss of offsite power; and (3) the capability of the system to transfer 4160 V safety buses to available power sources under conditions of postulated failures.
The team examined the adequacy of the size (kVA Rating) of transformers and the load capability (kM) of the EDGs.
The 4160 V Class 1E buses and connected cables and breakers were reviewed to assess load current capabilities.
The review also included the assessment of voltage drops in cables servinq safety-related loads to assure that acceptable starting and running term)nal voltages were available.
The team reviewed the adequacy of the overcurrent fault protection (short circuit and ground fault) and coordination of protective devices for the 4160 V system.
The electrical attributes examined included characteristics of the offsite grid with regard to its short circuit contribution to a fault in the EDS, and the fault current protection of the offsite source transformers.
The fault current, overload, and reverse power relay protection for the EDG were reviewed for adequacy.
The team reviewed the sizing of the load center transformers, circuit breakers, motor starters and buses.
The team also reviewed selected 480 V Class 1E motors for overcurrent protection, and terminal voltage including cable voltage drops'.during the starting and running of these motors.
This review included verifying the selection of the trip coils and motor overload heaters for the 480 V Class 1E motor-operated valves (MOVs), and the length and size of the cables connecting these MOVs to the motor control centers.
Electrical Distribution S stem Loadin The licensee presented Calculation 13-EC-MA-212, Revision 7, dated November 16, 1989, "Auxiliary System and Transformer Sizes," to demonstrate the'capacity of the station's auxiliary electrical distribution system including the Class 1E systems of the three units.
The team reviewed this calculation and identified the following concerns:
The worst case loading conditions of the preferred power supplies or startup transformers, with one of the three startup transformers not in service, failed to consider the non-Class lE loads being automatically transferred from the unit auxiliary transformer to the startup transformers and causing overloading of the startup transformer secondary winding.
Following a postulated loss of coolant (LOCA) initiated trip in one of the three units, and with one startup transformer out of service, the loading on one of the two secondary windings of a startup transformer could approach 105 MVA (58 MVA for a unit in startup mode and 47 MVA for a second unit in an accident condition).
The team also noted that a startup
I
0
transformer secondary winding could be overloaded if one unit was in hot shutdown with one train of its non-Class lE loads sharing the same startup transformer secondary winding as the unit in the postulated accident condition.
In this latter case, automatic transfer of the non-Class, lE loads was not a factor since these loads would have been previously transferred from the associated unit auxiliary transformer.
Calculation 13-EC-MA-212 determined that maximum potential emergency loadinq on Class lE, 480 V motor control center M-34 was approx>mately 240 kVA.
However, when the loading on load center L-34, which serves M-34, was determined, an incorrect value of 7.7 HP, or approximately 74 kVA, was used for motor control center M-34 contribution.
Calculation 13-EC-MA-212 did not consider transformer and cable losses in the calculation.
However, the team determined that the impact of the above listed errors on the total Class 1E bus loading was relatively insignificant.
The team was concerned that these calculation deficiencies in considering the worst case loading could affect the operation of the electrical distribution system equipment.
The team noted that these errors were originally a result of Bechtel efforts during the design of the units that had also been carried forward into plant operation.
The team also noted that, under the current program, Bechtel uses the licensee's design control program.
This program was reviewed to some extent during this inspection and the team considered that calculational errors such as the above should not occur under the current program.
The team also noted a discrepancy with various motor loading data used in Calculation 13-EC-MA-212 which was reported by the licensee to have been taken from Calculation 13-EC-PE-110, Revision 5, dated June 22, 1989,
"Diesel Generator Sizing."
However, the latter calculation had been superseded by Calculation 13-EC-DG-200, Revision 5, dated September 6, 1990, "Diesel Generator Load Calculation."
A spot check of several engineered safety feature loads indicated discrepancies between Calculation 13-EC-MA-212 and Calculation 13-EC-DG-200.
Thus, Calculation 13-EC-MA-212 had.not been revised to reflect the new load values (approximately 100 hp) used in Calculation 13-EC-DG-200.
As a result, personnel could have used the wrong load valves when referring to Calculation MA-212.
Procedure 81-DP-4CC04, Revision 1, "Calculations," requires that calculations be prepared in accordance with procedure 81-DP-OCC05.
Procedure 81-DP-OCC05, Revision 0, "Design and Technical Document Control," requires the responsible engineer to ensure that all affected engineering and plant configuration documents are addressed for update and to provide a design that meets all the functional requirements called for by the design change request.
The failure to reflect loads from Calculation 13-EC-DG-200 into Calculation 13-EC-MA-212 is considered an apparent violation (50-528/90-42-01).
f
2.2 Electrical Distribution S stem Volta e Re ulation The licensee presented Calculation 13-EC-MA-610, Revision 6, dated August 2, 1990, "Voltage Regulation Study" to demonstrate the capability of the electrical distribut)on system to provide adequate voltage conditions for Class 1E equipment under worst case system loading conditions and with the preferred power supply (the offsite 525 kV transmission'system)
at 95 percent nominal voltage level.
The team reviewed this calculation and noted that input loading data was taken from that developed in Calculation 13-EC-MA-212.
The team had expressed concerns with Calculation 13-EC-MA-212 as discussed in Section 2. 1, above.
Calculation 13-EC-NA-610 had considered that the worst case loading on the electrical distribution system and its impact on voltage regulation for a situation where the output of one startup transformer was not available, one unit was in startup mode, one unit was at full power operation, and the third unit in a LOCA condition.
The team concurred with this worst case consideration, but noted that the calculation failed to consider the impact of the non-Class lE loads being transferred to the startup transformers from the unit auxiliary transformer of the unit in an accident condition, followinq the trip of the accident unit.
Based on data taken from Calculation 13-EC-MA-212, the team noted that the transferred non-Class 1E loads are approximately
MVA for each of two redundant trains (reactor coolant pump motors, circulating water pump motors, condensate pump motors, etc.).
The loads would have been supplied, one each, in parallel with a redundant train of engineered safety features loads.
Under accident conditions, the team noted that the load demand of each train of engineered safety features loads would approach approximately 7 MVA.
The impact on voltage regulation on a startup transformer
MVA secondary winding by adding the relatively large block of 'non-Class lE loads (40 NVA per train) in parallel with the accident loads (7 NVA per train) and startup loads of the second unit (approximately
NVA per train) had not been analyzed.
The licensee performed a review of records (in response to the team's request)
to determine if there were actual periods of time in which the potential for overloading the startup transformers existed.
Records reviewed included maintenance records and clearance taps issued during the period of January 1988 to October 1990.
During thss period, the licensee determined that there were no loss of startup transformer incidents.
However, the potential for overloading existed during the following periods:
Between April 19 and April 21, 1988 and from April 28 to April 30, 1988, a potential existed for exceeding startup transformers X02 and X03 rating (Y and Z windings respectively).
Between August 9 and August 12, 1988, a potential existed for exceeding startup transformers X01 and X03 rating (Z and Y winding respectively).
Between January 19 and January 22, 1988, a potential existed for exceeding startup transformer XOl rating (Y winding),
Between May 13 and Nay 15, 1988, a potential existed for exceeding startup transformer X01 rating (Y winding).
I
Between April 14 and April 17, 1990, a potential existed for exceeding startup transformer X01 rating (Y winding).
CFR 50 Appendix A, Criterion 17, "Electric Power Systems,"
requires electric power from the transmission network to the onsite electric distribution system to be supplied by two physically independent circuits designed so as to minimize to the extent practical the likelihood of their simultaneous failure under operating and postulated accident conditions.
CFR Part 50, Appendix B, Criterion III, states that design controls shall be established to assure that applicable regulatory requirements and the design basis are correctly transmitted into specifications, drawings, procedures, and instructions.
Section 8.3. 1. l. 1 of the Palo Verde Nuclear Generating Station (PVNGS)
Updated Final Safety Analysis Report (UFSAR) states:
"Three startup transformers connected to the 525 kV switchyard are shared between Units 1, 2, and 3 and are connected to 13. 8 kV buses of the units.
Each startup transformer is capable of supplying lOOX of the startup or norma11y operating loads of one unit simultaneously with the engineered safety feature (ESF) loads associated with two load groups of another unit.
The non-Class lE ac buses normally are supplied through the unit auxiliary transformer, and the Class 1E buses normally are supplied through the startup transformers.
In the event of failure of the unit auxiliary transformer, turbine trip, or reactor trip, an automatic fast transfer of the 13. 8 kV buses to the startup transformers is initiated to provide power to the auxiliary loads."
Section 8. 1.4. 1 of the Palo Verde Nuclear Generating Station (PVNGS)
Updated Final Safety Analysis Report (UFSAR) states that one of the principal design basis applied to the offsite power system is that:
"The outage of a single startup transformer does not jeopardize continued plant operation, i.e., at least one offsite source to plant auxiliaries and ESF (engineered safety features)
buses is available with a single startup transformer outage."
P Calculation 13-EC-MA-212, Revisions 0 through 7 (Revision 7, dated November 16, 1989), "Auxiliary System and Transformer Sizes," did not consider the automatic fast bus transfer of non-class lE loads (approximately 40 MVA) from the unit auxiliary transformer to the startup transformer on a reactor trip, turbine trip, or a loss of coolant accident.
Consequently, the effects of this additional load on the ratings of the startup transformer windings and the evaluated voltages to engineered safety feature components (from reduced voltages)
were not considered.
Thus, the offsite power system was not designed to minimize the likelihood of simultaneous failure of both offsite sources when three Units are powered from two startup transformers and a fast bus transfer of non-Class lE loads from the unit auxiliary transformer to the startup transformer occurs.
For example, from August 9 to August 12, 1988, the ratings of startup transformers X01 and X03 (Z and Y output windings, respectively)
may have been exceeded due to excess current if a reactor
2.3 trip, turbine trip, or loss of coolant accident had occurred in Unit 2.
This may have resulted in a loss of both offsite power sources to Unit 2.
For example, from August 9 to August 12, 1988, the ratings of startup transformers XOl and X03 (Z and Y output windings respectively)
may have been exceeded due to excess current if a loss of coolant accident had occurred in Unit 2.
This may have resulted in a loss of both offsite power sources to Unit 2.
This is considered an apparent violation (50-528/90-42-02).
The team noted that the safety-related function of the Class lE electrical distribution system could have been supported by the standby EDG power system.
As a result of the team's identification of concerns regarding the loading of the preferred power source, the licensee performed a preliminary calculation using input loading data from the new worst case loading evaluation (addressed in Section 2. 1 of this report).
The preliminary calculation considered that the automatic transfer of the nonsafety-related loads from the unit auxiliary transformer was blocked (see Section 2.4).
The results of this calculation indicated acceptable bus and terminal voltages for safety-related motors during starting and running conditions with the preferred offsite supply voltage in a degraded condition of 95 percent of nominal (525 kV).
The calculation was based on a computer program,
"ELMS" that was reported to have been developed and certified by Sargent L Lundy Engineers.
The team reviewed the preliminary calculation inputs and results and noted that the minimum starting and running voltages were equal to or greater than the specified values for the safety-related motors.
Based on a limited review of this preliminary calculation, no immediate safety concerns were identified.
The team received the licensee's final calculation after the inspection period.
This calculation will. be reviewed by the NRC to assess its adequacy.
Protective Rela in The team reviewed the licensee's instruction document No. 7N431.02.01, Revision 1, dated March 3, 1990, "Protection, Metering, 8 Automated Control Instructions for Palo Verde Protective Device Settings,"
and noted a potentially misleading statement.
The instructions for several diesel generator protective relay setting procedures, stated that a
particular relay discussed "...trips the diesel generator only during test (non-loss of coolant accident condition)."
The team was concered this statement could be misinterpreted to imply that the tripping action for diesel generator loss of field, overcurrent, ground overcurrent, reverse power, etc.
was not blocked for loss of offsite power and auxiliary feedwater emergency actuation signal generations which was contrary to the station's UFSAR, Section 8.3.1.1.4.3.
The licensee agreed that a potential for misinterpretation existed and issued a request for revision to the procedure.
Also, during the team's review of the Calculation 13-EC-PB-100, Revision 2, dated April 15, 1985, "Class lE Metalclad Switchgear 4.16 kV Protection,"
a note was observed which referred to the inability to detect and isolate a phase to ground fault in the ungrounded portion of the 13.8 kV system during diesel generator testing.
Since the 13.8 kV system is grounded, the licensee reviewed the design and concurred that the note in question was in error.
The licensee stated that the note would be deleted when the calculation was revise.4 2.5 The team considered the above misleading and incorrect statements as two examples of a weakness in the licensee's measures established to assure that the design bases for the station were correctly translated into the design documents, procedures, and instructions.
However, there was no evidence that these misleading statements resulted in the introduction of errors into the plant.
Startu Transformer Loadin The team identified concerns with 'the potential for an inadvertent overload of a startup transformer secondary winding in the event that the output from one of the three startup transformers was unavailable (see Sections 2. 1 and 2.2 of this report).
The overloading would have resulted in increased voltage droop through the transformer and the potential for Class 1E bus voltage degradation.
In response to this concern, and to preclude the overloading, the licensee issued a procedure change notice, PCN No. 12, dated October 26, 1990, to the station operating procedure 430P-3NAOl, "13.8 kV Electrical System (NA),"
Revision 2, dated October 10, 1986.
The team reviewed the procedure change and noted that the revision instructed the operators that the
MVA (2929 amps at 13. 8 kV) startup transformer secondary windings would have sufficient capacity to assume a maximum load of 2641 amps maximum, leaving 288 amps (approximately 6.9 MVA at 13.8 kV) available for accident loads when needed.
The team found the procedure lacking in detail as to how the loading was to be monitored and controlled.
The 1icensee stated that the procedure was being clarified and further details and instructions were planned to be issued within five working days.
The final procedure apparently will involve blocking of the automatic transfer logic for 4he non-Class lE loads of the nuclear generating unit whose loads would have been conn'ected to their alternate startup transformer.
Nuclear safety-related loads of the two units, which would have been sharing a
common startup transformer winding, would remain unaffected.
The team considered such controls by the licensee appropriate.
The team considered that NRC follow-up of the implementation of this procedure should be performed to determine its adequacy (Open Item 50-528/90-42-03).
AC S stem Fault Anal sis The team reviewed the licensee's Calculation 13-EC-PK-620, Revision 4, dated November 27, 1989,
"AC Short Circuit Fault Calculations," to assess the adequacy of the inter ruptinq and momentary fault current rating of the Class 1E and non-Class lE distribution system equipment as applied at the station.
The document assumed that the voltage fluctuations on the utility grid were limited to 5 percent of the nominal voltage.
To verify licensee assumptions regarding grid voltage fluctuation, the team reviewed the licensee's voltage monitoring records over a period of one year.
The licensee's charts indicated that there were a few periods when voltage fluctuated up to 9.8 percent.
The licensee explained that these isolated cases of apparent voltage fluctuation were due to monitoring system errors and no actual voltage fluctuations had occurred.
In support of this, the licensee presented the system operator's log which indicated no system disturbances at the time of the anomaly.
The
O e
2.6 2.7 licensee also presented a record for a nearby switchyard recorder, directly tied to the Palo Verde 525 kV bus, which indicated a normal voltage.
The team assessed the above and considered the licensee's explanations acceptable.
In addition, the team compared calculated short circuit levels to the ratings of Class 1E breakers and found them acceptable.
Load Se uencin on Class 1E Bus The team noted that, during accident conditions, Class 1E loads are added to the bus using the automatic load sequencer.
The sequencer adds loads in seven steps at regular intervals of 5 + 1 seconds each.
The motor of each major load must start, accelerate to rated speed, and stabilize within this design interval of 5 + 1 seconds.
The same sequence and timing is used whether power is supplied from the preferred or standby source.
The team was concerned that under design basis degraded voltage conditions at the Class lE bus (when connected to the preferred offsite source),
a major pump motor may not reach the rated speed and stabilize before the sequencer initiates the start of another motor load.
The team discussed this concern with the licensee who indicated that they had performed a short calculation based upon vendor motor data for the largest sequenced motor.
The calculation and the vendor data revealed that the motor may not attain rated speed even after 4 seconds during degraded voltage conditions.
The team then became concerned that the condition may further deteriorate if a second motor was started within this interval.
During this inspection, the licensee performed a preliminary voltage regulation study (see Section 2.2 of this report)
and demonstrated that the sequenced starting of motors with an overlap of the largest motor and the next motor in'sequence would not compromise the starting of the motors.
Acceptable starting voltages were demonstrated.
Diesel Generator Loadin The team noted that two emergency diesel generators provide "onsite" power for Class 1E buses.
Each generator is rated at 5500 kM at 0.8 power factor, and meets 100 percent of the demand of one train of engineered safeguards equipment.
The team found that calculation 13-EC-DG-200, Revision 4, dated September 6, 1990, "Diesel Generator Load
.
Calculation," tabulated the loads for Class 1E buses.
The team noted that in one case (forced shutdown),
the margin between the loading and generator rating was below 1 percent.
Loading did not include the transformer losses which amounted to approximately
kM.
The listing of the motor loads was based upon the BHP demand of the load such as pump, compressor, or fan and not on the nameplate rating of the motors.
The team raised a concern that any undetected mechanical problem with the load could raise the demand on the motor at least up to the extent of its full load rating.
The licensee stated that conservatism was used in listing loads such as panels, heaters, battery chargers, etc.
The
2.8 2.9 licensee also explained that the subject document was under revision and the next revision would present a better analysis.
Based on the informa-tion provided by the licensee, the team had no immediate safety concerns.
However, the team considered that a follow-up NRC review should be conducted of the licensee's revised DG'oading analysis to ensure its adequacy (Open Item 50-528/90-42-04).
Class lE DC S stem Batteries and Char ers Calculation 13-EC-PK-202, Revision 2, dated August 22, 1990,
"2-Hour Battery Calculation," included DC loads on each battery bus, a load profile for two hour loading, and calculation for the sizing of the batteries and the battery chargers.
The team observed that the calculations did not include any design margin to account for future growth.
The referenced standard, IEEE 485, in the calculation called for a 10-15 percent design margin to allow for future load addition, any recent discharge, and improper maintenance.
The calculation also indicated that there was approximately a four percent margin available on the batteries.
The licensee explained that they considered the maintenance program in place was adequate and that the capacity of the battery would be evaluated before adding any additional load.
The team considered this acceptable.
DC S stem Fault Anal sis 2.10 The team found that Calculation 13-EC-PK-110, Revision 3, dated November 16, 1989,
"Class lE DC Short Circuit Calculation," provided insufficient information since":it did not include cable length, equipment vendor data, or a complete list of references.
The licensee took immediate action to revise the calculation during the inspection period.
The team reviewed the revised document, Revision 4, dated October 24, 1990 and considered it acceptable.
Conclusions e
Based on the insp'ection sample, the team did not identify any operability problems, and concluded that the design.of the EDS system was generally adequate.
The major concern identified by the team was the apparent inability of the startup transformers to meet GDC 17 requirements under all possible scenerios.
In addition some deficiencies existed with calculations that were performed by Bechtel, including unanalyzed postulated conditions of operation, unsubstantiated assumptions, and an incorrect statement.
The team had no significant safety concerns because of additional analyses and information provided by the licensee during the inspection.
Emergency power sources were sized properly, and adequate voltage was appl)ed to essential buses with regard to EDS loads.
The protection and coordination of protective devices applied in the EDS equipment to protect the system and engineered safeguards equipment motors were
3.0 3.1 adequate.
Staff support for the EDS from the corporate office and site was adequate.
Staff were knowledgeable and competent.
Mithin this area inspected, two violations were identified.
MECHANICAL SYSTEMS To determine the functional ability of mechanical systems to support the EDGs during postulated design basis accidents, the team reviewed sample documentation and conducted walkdowns of fuel oil storage and transfer, lubricating oil starting air, and diesel heating and cooling equipment.
The team reviewed equipment associated with the heating, ventilating, and air conditioning (HVAC) of the diesel generator building, battery rooms, essential switchgear rooms, and selected EDG and HVAC design modifica-tions.
The team also reviewed the translation of various mechanical loads (selected pumps/MOVs) to electrical loads for input into design basis calculations.
Essential S ra Pond S stem The team noted that the Unit j. Train A spray pond piping, supports and heat exchanger nozzles had experienced waterhammers in the past.
The licensee made a modification to the system to try to correct the problem by adding a vent line, but the team considered that the testing that was performed was not adequate to demonstrate that the waterhammer was eliminated.
For example, it appeared that a waterhammer could take place if the pump was tripped and restarted within a minute since there could still be a vacuum below the pump and its support plate.
The testing performed was done en the B Train pump (with a'similar modification) which had never experienced a problem with waterhammer in the past.
In this test, the pump was poised for approximately two minutes before r estart, allowing adequate time for pressure equalization.
Calculation 13-MC-SP-302 Revision 1, dated March 29, 1988, "Hydraulic Transient Ana]ysis of the Essential,"
simply stated that three pump quick restart tests were performed on Train A with a temporary 1-inch diameter vent line installed to verify the proposed fix and that all tests verified that the waterhammer had been eliminated.
However, there was no supporting documentation.
Subsequent to the team's finding, the licensee performed a calculation to analyze the effects of waterhammer on the piping, heat exchanger nozzles and piping supports during quick pump restart.
The resulting stress analysis showed that the pipe stresses were within ASME code requirements and heat exchanger nozzle loads were within specification allowables.
Pipe support loads were also within acceptable limits.
The licensee stated that this effort will be included in Calculation 13-MC-SP-505.
During walkdown of the spray pond system, the team noted that the vent lines for the pumps were under the water in the pond.
The team was concerned that this would defeat the intended purpose of the vent line to be used as a vacuum breaker.
The licensee reviewed the problem and
3.2 determined that it was due to a drawing error.
The installation drawing 13-P-XYS-062, Revision 17, "Essential Spray Pond Piping Sections and Details," indicated an incorrect spray pond level to which the vent line had been installed.
The spray pond level could in fact be higher than what was indicated on the drawing.
In addition, it was apparent that there was no post maintenance test (PMT) or walkdown of the modification as a follow-up to assess the adequacy of the installation.
For corrective action, the licensee planned to modify the vent line to ensure that it is above the high water level (worst case scenario)
as originally intended in the Design Change Package (DCP)
No.
2SM, 3CM-SP-040, Revision 0.
The licensee also planned to correct the discrepancy between the isometric Drawing 13-P-ZYA-062 and the level Setting Diagram 13-J-03K-057, Revision 6, "Essential Spray Pond Level Setting Diagram," by issuing a procedure change request for Drawing 13-P-ZYA-062.
Fuel Oil Stora e and Transfer S stem Fuel Oil Da Tank Vent The team noted that the vents for the diesel generator fuel oil day tanks were seismic category 3.
However, there was no analysis for postulated tornado wind load>ng and missile impacting of the vent lines even though the top seven feet of the vent lines were located outside the building.
The licensee took immediate action to evaluate this condition and determined that the resulting stresses due to the application of tornado wind speed to the existing p)ping system (vent line above roof) were within ASME code requirements.'.
This was documented in revisions to Calculations 13-MC-DF-501 and f3-MC-DF-502.
The licensee'lso addressed the effect of missile impact on the vent line.
The vent line had not been analyzed or designed to withstand this event.
However, the licensee indicated that the consequences of the pinching the vent line closed, or the shearing off the vent line were minimal.
The licensee stated that the. pinching of the vent line would not be a
safety concern since a venting route was still available via the fuel day tank overflow line to the fuel oil storage tank.
If the vent line was sheared off, water or dirt could enter the fuel system.
The licensee analyzed the number of inches of rain per year, the unusable fuel at the bottom of the day tank and the procedure to drain the day tank to remove any condensation which may have accumulated to demonstrate that the day tank fuel would not be compromised.
The team had no further concerns in this area.
Fuel Oil Stora e Tank Level Calculation The team reviewed Calculation Number 13-MC-DF-306, Revision 2, dated September 25, 1990, "As-Built Calculation for Sizing Diesel Fuel Storage and Diesel Generator Day Tanks."
The team noted that the calculation used a conservative fuel consumption rate of 6.5 gpm, compared to vendor data which ranged from 5. 94 gpm to 6. 06 gpm.
In addition, the onsite as-built test data was approximately 6.4 cpm.
However, the team did not find any documentation of the actual specific gravity or the temperature
1 li
3.3 of the fuel for this as-built test.
The team noted that the fuel consumption rate could vary due to these parameters.
The team found that the sizing calculation for the storage tank accounted for the unusable fuel volume, the required fuel level to meet 7 day usable supply for rated capacity of, 5.5 NM, a 15K allowance for periodic testing, and a margin.
The team found that the unusable fuel volume was 4600 gallons and not the 4463 gallons indicated in the calculation.
However, there was sufficient conservatism in the fuel consumption rate and margin to cover the error in the unusable volume.
The error was significant only from the standpoint that APS had made several revisions to the calculation (most recently September, 1990)
and had not identified this error that dated back to the original calculation.
The inspector noted that the assumed unusable fuel level was 16.5" from the bottom of the tank and was sufficient to account for the fuel displacement by the pump and any vortex associated with the pump suction.
In addition, the pump suction was approximately 6.5" from the bottom of the tank and facing downward.
The day tank sizing included one hour of fuel supply, a 10X reserve margin, and an unusable fuel volume.
The transfer pump was energized to add fuel to the day tank to ensure adequate fuel above the technical specification level.
The team noted that although there was no level setting diagram for the storage tank, the instrument setpoints for EOG fuel transfer appeared to be adequate.
The diesel fuel transfer and storage system appeared to meet the TS requirements of maintaining a minimum level of 2.75 feet (550 gallons of fuel) in the day tank and 71,500 gallons of fuel in the storage tank.
Essential HVAC The team noted inconsistencies between the Updated Final Safety Report, (UFSAR), Chapter 9, Table 9.4. 2, and Calculation 13-MC-HJ-252 Revision 1,
'ESF Switchgear Area Essential Equipment Sizing Calculations," for the room design temperatures with regard to service conditions.
For example, the UFSAR, Table 9.4.2 indicated that the temperature in the ESF switchgear room was required to be maintained between 60'F and '77'F during essential HVAC operation whereas the design criteria manual required a temperature range of 40'F to 104'F.
In response to team's findings, the licensee reviewed these inconsistencies and initiated an engineering evaluation request (EER) for correcting the affected documents.
The team further asked the licensee to determine if there were any inconsistencies between the various design basis documents for normal HVAC operation.
The licensee's response was that there were no inconsistencies but that there was an error in the design criteria manual for essential battery rooms which indicated a normal temperature range of 60'F to 77'F instead of the correct range of 60'F to 85'F.
The licensee further indicated that the design criteria manual will eventually be replaced by the design basis manual and correction will be carried out at that time.
This assessment was accepted by the tea.4 Conclusions
~
~
The team concluded that the appropriate technical staff were knowledgeable of their mechanical systems affecting the EDS.
Sufficient information was available to review and assess the operability of these mechanical systems.
As a result, the team considered this a strength in regard to engineering and technical support.
A number of findings were identified in.the mechanical area with regard to inadequate design and calculati'on reviews which indicated a weakness in the design control of EDG and associated equipment.
However, the team had no concerns regarding the operability of this equipment.
4.0 Within this area inspected, no violations were identified.
ELECTRICAL DISTRIBUTION SYSTEM E UIPMENT To'confirm the implementation of the electrical system design, the team inspected the as-built configuration of selected safety related equipment and performed a review of testing and surveillance procedures related to the UGs, batteries, inverters, switchgear, circuit breakers, fuses, and relays.
The team also examined a limited number of design change packages and site modifications, including the design review and approval process and related safety evaluations in accordance with 10 CFR 50.59.
~
~
~
~
The specific areas of review are discussed below.
4. 1 E ui ment Walkdowns The EDS equipment was reviewed. to assess its conformance to the design drawings and requirements.
The distribution equipment, including high, medium, and low voltage switchgear, load centers, motor control centers, and panels, were found to be properly labeled, easily identifiable and accessible.
Proper physical separation existed between redundant buses, equipment, and components.
Separation was accomplished by distance and segregation of equipment in separate fire zones.
In addition, housekeeping was found to be generally adequate for all.three units with few deficiency tags.
Drawings used for the walkdowns were clear, and in most cases reflected the as-built configurations.
Only minor inconsistencies with no safety significance were observed between design.drawings and installed equipment.
Confi uration Control Equipment configuration control appeared to be adequate.
However, two minor concerns were identified with respect to missing Kirk key locks and transformer tap change restraints.
In the first case, the team observed that some Kirk key locks associated with the control enclosures of the ESF and unit auxiliary transformers were either missing or disabled.
It was later determined that the locks had no interlocking function and that an engineering evaluation had been performed to replace
4 J'
them with equivalent locks from a different manufacturer.
However, the actual removal of the locks in Unit 2 took place in July, 1989 and no documentation existed to determine when other locks would be installed.
The concerns expressed with respect to this issue were the obvious deviation from the EER recommendation and a potential inadvertent or unauthorized repositioning (to OFF) of the transformer fans'ontrol switches.
This could change the actual kVA loading of the transformers in question.
With respect to the transformer tap changers, the team observed some inconsistencies in the use of padlocks.
Therefore, the licensee was asked if procedures existed to control such use.
The team was later informed that no procedures existed, but that padlocks would be installed, where missing.
Before the end of the inspection, the licensee issued work requests (WRs) 778764 and 778765, to install the missing padlocks.
b
~L'1 T
During walkdown of class lE cable installations the team noted a number of locations where tie wraps were not installed every 5 feet in vertical trays as required by licensee specification 13-EN-300, "Installation Specification for Electrical Cables in Cable Trays," Revision 1, August 28, 1987.
The licensee stated that the tie wraps were used to secure the cables to the trays and not for vertical support.
The team noted that IEEE Std. 690-1984,
"IEEE Standard for the Design and Installation of Cable Systems for Class 1E Circuits in Nuclear"Power Generator Stations,"
recommended in paragraph A9.4(ll) to secure cables to their trays every
to 5 feet in vertical trays.
Procedure 13-EN-300 was in agreement with the IEEE,recommendations.
The team discussed this concern with the licensee, who evaluated this condition and determined that the lack of supports for cables was of no'consequence to the integrity of the cables.
However, the licensee initiate'd work requests to install the tie wraps as required by procedure 13-EN-300, to better secure the cables.
The team considered that this lack of conformance to the procedure was of low safety significance.
This is a non-cited violation since the criteria specified in Section V.G of the Enforcement Policy were satisfied and this item is closed (50-528/90-42-05).
Emer enc Diesel Generator EDG) and the Associated Su ort S stems During a plant wa1kdown, the team raised a concern with the potential for plugging of the flame arrester vent on the diesel fuel oil storage tank and inquired about any existing maintenance program for cleaning the vent.
The licensee indicated that there were currently no preventive maintenance tasks for cleaning these devices.
However, as a result of the team's concern, an instruction change request, Number 25059, was issued to develop such a task.
The team noted that the present ASNE Section XI testing for the fuel oil transfer pumps would detect any degradation in pump performance caused by the vent plugging.
As such, the team did not consider it a problem.
During another plant walkdown, the team observed the lack of low point moisture drains in diesel air start piping.
Since this issue was corrected by plant change request 90-13-DG-031 (as mentioned in the
4.2 Equi pment Node ficati on Section of this repor t -Paragraph 4. 2) the team considered this issue resolved.
E ui ment Hodifications The team reviewed the current program and found that plant modifications were processed either through the site nuclear engineering organization or through the nuclear engineering department.
In general, only non-complex modifications, which have no impact on the design basis of the plant, were processed through'site nuclear engineering.
Separate procedures were used for the two types of modifications.
The current engineerinq design and modification control process was found to be well proceduralized.
However, there were no modifications implemented under the new program for review by the NRC to assess the adequacy of the current program.
The team reviewed several design changes that were implemented under licensee's previous design change programs.
The team walked down portions of 3 plant design changes involving 125 VDC and 120 VAC safety equipment.
In addition, the team walked down portions of 3 plant design changes involving installation of new safety related cables, and changes to the emergency diesel generators and associated systems.
With regard to the electrical equipment and cables, those sampled for inspection were installed in accordance with the design change package instructions.
The team also noted that associated plant drawings were properly updated to include the changes.
The team reviewed several older plant change packages for the EDGs and associated systems and found rehem to be adequately implemented.
Examples were plant change package 85-01-DF-004-00, which added a graduated sight glass on the diesel fuel oil day tanks by modifying the existing level switch standpipe, and plant change package 84-01-DF-002-00, which changed level setpoints for diesel fuel day tanks to ensure that the diesel fuel transfer pump cycles to maintain level above the TS required level of 2.75 feet.
The team also reviewed plant change request 90-13-DG-031, which had been approved for implementation, for replacing diesel startinq air compressors (4 per unit) and dryers (4 per unit), relocating the compressor discharge relief and check valves, replacing present moisture traps, adding low point drains, and rerouting associated piping.
This change was in response to earlier regulatory concerns.
The team considered this plant change request to be adequate.
These design changes were reviewed and approved in accordance with the TS and the established gA/gC controls.
In addition, post modification tests were conducted and test result evaluations were performed before declarinq the affected systems and components operable.
Test results were reviewed against previously established acceptance criteria.
The records of modifications reviewed were generally complete and comprehensive.
The team also considered that the design evaluation process appeared to effectively incorporate
CFR 50.59 safety evaluations (as applicable) for both types of modification.3 One modification where the team identified a relatively significant problem involved the spray pond system as discussed previously (in Paragraph 3. 1) of this report.
E ui ment Maintenance Testin and Calibration The team reviewed the licensee's program for periodic testing of electrical equipment; including, emergency diesel generators and associated systems, batteries, battery chargers, inverters, medium and low voltage circuit breakers, control and protective relays, and motors.
Evaluation of this program indicated that the licensee has developed the necessary procedures to address testing of equipment which is identified in the Technical Specifications.
The licensee had recognized that numerous problems existed in the Palo Verde maintenance program for safety related equipment.
As a result, they created a preventive maintenance task force to develop a
new maintenance program partially based on reliability centered maintenance concepts.
The team reviewed preliminary samples of the licensee's new maintenance procedures and found the samples contained adequate maintenance requirements.
The team also noted an adequate management commitment to provide dedicated resources to the new maintenance program.
However, the team reviewed the maintenance and testing program for the all AC voltages and 125 VDC Class lE systems and considered that a number of weaknesses existed.
Switch ear Maintenance The team noted that the licensee was not performing routine internal inspection and cleaning of Vital Instrumentation and Control Distribution Panels lE-PNA-D25, 1E-PNB-D26, 1E-PNC-D27 and lE-PND-D28 in Unit l.
(These panels distribute class lE 120 YAC power.)
The licensee stated that no routine internal inspection and cleaning had been done on these panels in unit 1 since 1984, in unit 2 since 1985 and in unit 3 since 1987, except for panel D25 an unit 2.
Panel D25 in Unit 2 was cleaned and inspected in May 1990.
The team noted that two preventive maintenance (PM) tasks applied to these panels, a minor inspection every 12 weeks and a more detailed inspection and cleaning once every three refuelings.
However, the 12 week task was not authorized and both tasks were scheduled for 1999 on the licensee'.s PM list.
The licensee reported that unscheduled tasks were listed as scheduled for 1999 to complete a
computer required field.
The vendor technical manual, Palo Verde document PV-E020-83, General Electric GEK-71978, Revision 0, recommended a yearly inspection as the minimum requirement.
Due to the design of these panels the licensee considered it unsafe to perform an internal inspection with the panels energized; therefore, the team was unable to determine the internal condition of the panels.
Technical Specifiction 6.8. 1 and Appendix A of Regulatory Guide 1.33 prescribe that maintenance affecting safety related equipment be preplanned and performed in accordance with appropriate written procedures/instructions.
Failure to perform preventive maintenance for these safety related panels is considered an apparent violation (50-528/90-42-06).
0
As a result of the team's concern, the licensee issued instruction change request number 22682 to reinstate periodic internal inspection and cleaning of these panels.
~
'ircuit Breaker Testin Evaluation of licensee's test program revealed that the General Electric (GE), type AKR, 125 VDC circuit breakers were tested using a multi-ampere AC current source.
The team questioned the accuracy of testing DC circuit breakers with an AC current source.
The licensee indicated that they had contacted the manufacturer and addressed the issue in EER 83-PK-002, dated October 3, 1983.
Review of this EER revealed that in a letter dated August 15, 1983, GE had informed the licensee that the subject breakers responded differently to the same magnitude of AC and DC current applications and that a correlation factor between AC and DC currents had never been developed.
Therefore, GE explained that "any calibration of the AKR breakers must be done with DC current."
In a subsequent letter, dated November 4, 1983, GE recommended either the establishment of a baseline using an AC current source, or the creation of an AC/DC correlation for each breaker.
The original resolution of the EER accepted GE's recommendations.
However, a later evaluation of the issue by the licensee concluded, on December 7, 1983, that an AC test current source was acceptable and that a conversion factor was not required for AC current.
The bases for the conclusion:
(1)
GE has already calibrated the breakers; (2) trip devices respond the same for instantaneous AC and DC currents, (3) frequencies from 0 to 120 Hz can be applied to AKR breakers; and (4)
a correction factor was used by only one of nine plants which use an AC source.
In response to the team's concern regarding this issue, the licensee produced a Pennsylvania Power a'nd Light (PP8 L) study that documented the testing performed by a contractor for a similar NRC concern with breakers installed at Susquehana Nuclear Station.
The results of that study indicated that the DC breakers operated satisfactorily within the published GE time current characteristics using an AC source.
This study was documented in PP8 L Engineering Study EE-271.
In addition, the licensee indicated that they will evaluate further the need to change the testing method for these breakers.
The team considered the licensee's proposed action adequate.
.
Regarding the use of an AC source for molded case circuit breakers which used thermal trip devices, the breaker manufacturer confirmed acceptability of an AC source in lieu of a DC current source.
Another concern noted with the licensee's program for surveillance testing of medium and low voltage circuit breakers was that it did not address all Class 1E breakers.
For instance, no specific procedure existed for testing the 4. 16 KV safety related load center breakers that were not included in the TS.
The same was true for 480 V load center breakers which were not scheduled for a specific surveillance test, except in the case of motor feeders which were exercised with the respective pumps.
Therefore, motor control center feeders were tested only in conjunction with maintenance activities.
The team also noted
0 that several safety related molded case circuit breakers were not included in any testing program.
This concern was discussed with the licensee who indicated that they planned to include safety related molded case circuit breakers in a testing program.
Protective Rela Testin-The team verified that the licensee had adequate procedures for calibrating protective relays and inquired about the frequency of this activity.
The team determined that, in accordance with commitments made to the NRC, (in a letter dated October 5, 1984), the licensee calibrated the undervoltage protective relays for Class IE 4. 16 KV switchgear and the diesel generator protective relays every refueling outage.
However, according to the commitments of the same letter, they calibrated the remaining relays, including IFC overcurrent relays, every third refueling outage.
The manufacturer's recommendations for type IFC overcurrent relays was a testing frequency of one to two years until the user accumulated enough experience to select the interval best suited for his individual requirements.
Discussions with the licensee and evaluation of previous test data indicated that no attempt was made to record as-found data and, thus, accumulate the necessary experience with the relay to establish the calibration schedule.
As such, there was no documented justification for performing testing of these relays at the current frequency.
The licensee was currently in the process of updating procedures and more recent revisions required the. recording of as found data.
However, the lack of data relative to protective relays, such as the IFC, was a
concern in that without records.- of drift the licensee could not be sure when a particular relay was going to be out of tolerance.
Adequate calibration of these relays was )mportant to ensure that proper coordination existed between trip devices of main and feeder breakers and that the single failure criterion was not violated.
This item will remain unresolved pending the licensee's evaluation of the significance o'f testing relays in this manner and frequency (Open Item 50-528/90-42-07).
Notor Testin e
Another example of a testing weakness the team observed was that no procedure existed to routinely measure insulation resistance of safety related pump motors.
IEEE Std, 43-1961 recommended performing 1 and
minute insulation resistance tests to determine and trend motor conditions.
The ratio of the two tests was defined as a polarization index.
Although the licensee was not committed to IEEE Std. 43-1961, the team considered that polarization index data was useful'n assessing motor conditions.
The licensee agreed with the concern and indicated that they will include routine motor insulation resistance testing and evaluation in future procedure Diesel Generator Testin Discussions with the licensee indicated that during certain events which caused the automatic starting of the diesel, if the offsite power was available, the diesel was allowed to operate unloaded for an indeterminate period of time.
This was not usually recommended by the diesel manufacturers and could result in soot build-up, requiring the need for running the diesel loaded for a period of time after it has run unloaded.
The licensee also provided a copy of the applicable procedure.
The team noted that the licensee had a procedure, 41AO-lZZ52, for operating the EDG at load after running unloaded for most conditions.
However, the NRC noted that the current procedure did not require the restart of the diesel in the event of a loss of offsite power, during the period between shutting down and restarting the diesel.
The licensee agreed with the 'team's concern and initiated a procedure revision which required the loading of the diesel prior to its shut down.
EDG Su ort S stem Calibration The team reviewed the calibration program for level instrumentation on EDG fuel oil storage and transfer system and found that it did not account for the variations of temperature and specific gravity of the fuel.
The level instrumentation was calibrated at whatever the temperature and specific gravity of the fuel was at the time of calibration.
The two parameters were not recorded at the time of calibration, and adjustment was not made to accommodate the variations in the calibrated data due to the effects of the variations in these parameters.
The team noted that the temperature of the ground at the diesel fuel oil storage tank ranged from approximately 60 to 80 'F.
The day tank room temperature was'.controlled from 50 to 122 'F.
In addition, the specific gravity of the batches of delivered fuel oil ranged from 0.852 (July 24, 1989) to 0.871'(August 8, 1989).
Thus, the calibrations performed did'not account for these variations of temperature and specific gravity.
The team also reviewed some preventive maintenance records which indicated difficulties in keeping the level instruments within tolerances.
EDG Start Circuit Rela Testin On October 30, 1990, at a~proximately 6:21 a.m.,
Diesel Generator, 3A received a "High Priority'rip signal while the generator was paralleled to the offsite power source.
The EDG had been loaded to 100K (5500 KW)
for approximately 10 minutes when the trip signal was received.
The
"EDG A High Priority TRBL" and
"EDG A Trip" annunciations were received in the main control room.
The on-,shift operator responsible for the EDG immediately noticed that the EDG output power was 0; however, the EDG A output breaker indicated closed.
In addition, EDG A voltage and frequency meters were still reading 4160 VAC and 60 hertz, respectively, and the EDG reactive load indicated approximately 4000 KYAR, leading.
As a result, the operator tripped the breaker open causing the EDG to coast down to a stop.
An auxiliary operator was dispatched to the EDG A Control Room and he reported that the "Excessive Vibration Engine" and
"Field Loss" local annunciator windows were illuminated.
The EDG was secured by the operations staff declared "inoperable",
and Engineering was notified of the event immediately.
Problem review report 0001042 was
'initiated to investigate the cause of the event.
The troubleshooting effort indicated that trip signal relay '52T2'n the EDG engine control cabinet would not cycle when energized.
In addition, trip s)gnal relay
'52T1'ad high resistance across one set of its contacts.
Troubleshooting revealed that the prot'ective relays operated properly and remained functional and the generator field was not grounded or open-circuited.
As a result, the licensee concluded that the generator had sustained no apparent physical damage.
The team noted that; normally, relay '52T2'long with relay
'52T1'unction to provide a trip signal to the DG output breaker, 3EPBAS03B, when the EDG is shutdown either by the handswitch or the high priority trip.
However, in this event, these two safety-related relays responsible to trip the diesel generator output breaker failed to function as designed.
The '52T2'elay was tested and found to have an open coil, and the
"52Tl" was found with one of its four contacts reading higher than normal resistance reading.
The team noted that these two relays were not individually tested, nor were they included in the preventive maintenance program.
After further inquiry by team, the licensee identified that one additional safety related=relay in the EDG generator circuitry, relay '3GUP2',
was not included in any surveillance testing"or PM task of the EDG.
The failure of this relay would prevent parallel) ng of the EDG to the bus.
The team discussed with the licensee the concern that there was a lack of individual preventive maintenance and testing for safety related relays in order to trend their performance and to predict impending failures.
Thus, the first indication of 'a problem was when the relay actually failed.
The licensee indicated that they would evaluate the team's concern.
Other EDS E ui ment Testin The team noted that no periodic test was performed to check the battery charger DC overvoltage, DC undervoltage, reverse current and AC undervoltage set points and alarm relays.
The team also noted that no periodic test was performed to check 120 VAC Class 1E inverter DC undervoltage, AC overvoltage and AC overcurrent set points and alarm relays.
The team considered that some testing or calibration of these devices may be warranted.
The licensee evaluated this concern and agreed to consider periodic testing for some of these devices.
maintenance and Surveillance Procedures The team found that the licensee has 'been upgrading maintenance and surveillance procedures and a number of improvements in EDS maintenance and surveillance procedures were observed with those issued within the
0
last year (as compared to previous years).
However, some minor weaknesses were noted with procedures.
The team reviewed a number of EDS maintenance and surveillance procedures and noted two minor weaknesses; lack of acceptance criteria and lack of criteria for test equipment.
The team considered that these procedures could be enhanced by providing clear criteria to craft personnel since the completed procedures were not normally reviewed by engineering unless problems were identified.
For example, Palo Verde maintenance procedure 32MT-9ZZ58, "Maintenance of Inverters," Revision 1, Procedure Change Notice (PCN) 2, dated August 22, 1989, specified in paragraph 8. 2. 2 to use appropriate test equipment and verify that seven meters were reading correctly.
However, no acceptance criteria was listed and the test equipment was not defined.
Another example was with Palo Verde surveillance procedure 32ST-9ZZ74
"Molded Case Circuit Breaker Surveillance Test," Revision 3, PCN 1, dated January 18, 1990.
It specified in paragraph 8.4 to measure and record breaker contact resistance.
However, no acceptance criterion was given.
The team noted other examples of EDS procedures which required recording data without acceptance criteria or that did not specify test equipment to be used.
The licensee's maintenance procedure organization reviewed the specific examples with the team and agreed that the engineering organization preparing the procedures should specify the required acceptance criteria and test equipment.
The licensee also noted that some data was for information only and required no acceptance criteria.
The licensee will include acceptance criteria for test equipment and data or note that the data is for information only in future procedure revisions.
Y Observation of Maintenance Surveillance The team observed part of surveillance, 31ST 9DG01, "Diesel Engine
Month Inspection," being performed on the Unit 3B EDG.
The surveillance appeared to be conducted in an adequate manner.
Preventive Maintenance Task Force Pro ram The team noted that the reliability centered maintenance program was suspended, but the data gathered was being used by the PMTF program.
Overdue Preventive Maintenance Items The team reviewed the overdue surveillance report dated September 28, 1990, and observed 13 overdue electrical items for Unit 1, none for Unit 2,
and 14 overdue electrical items for Unit The licensee ran a query for all overdue preventive maintenance (PM) for the diesel generators, diesel fuel oil, and diesel generator e'lectrical parts.
One item from Unit 3 was identified overdue past the 25K grace period. 'his item was associated with WR 057976 for the lubrication of seal rings and calibration of a switch.
The work was last performed on June, 1989.
Other than this, PM items appeared to be performed within the required intervals.
Overall Performance of Electrical Maintenance The team observed the following statistics for Unit 3 electrical maintenance for two periods to determine if a backlog existed.
The following statistics were noted:
Received work requests Cancelled work requests Issued work orders Completed work orders Closed work orders Period of 1/90-11/90 1284 226 3460 2554 2817 Period of 8/90-11/90 321
1010
'55 1129 The team noted that the number of work requests received per month for the period of August to November 1990 (approximately 100) were about the same as for earlier months in 1990.
However, the number of work orders issued increased in relation to earlier months with approximately 30K of the work orders being issued for the period of August to November 1990.
In addition, the number of closed work orders increased in relation to earlier months with approximately 40K of the work orders being closed for the period of August to November 1990.
Thus, the team concluded that the workload of electrical maintenance at Unit 3 was still relatively large, but the backlog was slowly being reduced by the licensee.
Fuse Control The team reviewed the licensee's control of fuses and found that there was no documented fuse control program.
The licensee indicated that single line diagrams were kept accurate
.to show fuse current ratings.
The remaining fuse information, such as style, type and voltage rating could be found in a controlled copy of the appropriate vendor technical manual.
The team walked through replacement of fuses with two separate licensee planners and noted that both followed the informal licensee instructions.
However, the team noted that the licensee did not keep the vendor technical manuals up to date for licensee initiated changes dealing with fuse current ratings.
For example, licensee single line diagram, Ol-E-PNA-002, "120 VAC Class Power System Ungrounded Vital Instrument and Control Distribution Panels 1E-PNB-D26 and lE-PND-D28,"
Revision 6, indicated fuse D2824 was a 15 amp fuse.
The vendor technical manual, PV-E020-83, indicated fuse D2824 was a six amp fuse.
Since personnel responsible for replacing fuses had to use selected information from vendor technical manuals that did not contain all the correct fuse
5.0 information, the team considered the lack of a written specific fuse control program could lead to mistakes and incorrect fuses being installed.
The team performed a review of recent EDS work requests and quality assurance records and found that they did not produce any examples of incorrectly installed fuses.
The licensee indicated that they intended to eventually include all fuse information in a computer based ma'terial control program.
The team observed several EDS fuses and compared these fuses to the system one line diagrams and vendor technical manuals.
No problems were noted.
The team also noted that the installed fuses had proper voltage ratings to provide interrupt capacity for maximum faults.
The team reviewed the voltage and current ratings of selected additional fuses, and concluded that, for the selected sample, the fuse voltage and current ratings were adequate.
Conclusions The team concluded that the maintenance and surveillance programs were adequate with regards to meeting TS requirements.
However, there were a
number of concerns related to testing of other safety related EDS components and the adequacy of procedures in relation to the use of acceptance criteria.
The team considered that the licensee was responsive to the team's concerns and actions were underway in response to those concerns.
Previously, APS performed a number of root cause evaluations for events such as transformer problems, an electrical systems reliability study, and an electrical systems assessment.
In these reviews, APS noted that there were problems in high voltage equipment that could be attributed to maintenance/testing deficiencies.
The licensee appeared to have satisfactorily resolved those particular issues and has started to implement a
PM task force to address weaknesses in this area.
However, the team noted that some of the items reviewed by the team were not yet being considered for inclusion into any PM or surveillance program and the RCM effort had been suspended.
Thus, the team considered that APS may want to continue to challenge the assumptions made in preparation of the PM effort.
One violation and one non-cited violation were identified.
En ineerin and Technical Su ort The team assessed the capability and performance of the licensee's organization to provide engineering and technical support (E8TS).
The team examined interfaces between the technical disciplines internal to the engineering organization and between the engineering organization and the functional groups performing design reviews, field modifications, surveillance, testing, and maintenance.
The team also reviewed a sampling of licensee event reports (LERs),
non-conformance reports (NCRs), Independent Safety Engineerinq Group (ISEG) operation event reports (OERs),
and recent NRC inspection reports to attempt to ascertain the.adequacy of the licensee's root cause analysis and corrective action program A (
0 5.1 The team noted that the licensee has been implementing a number of organizational changes and program enhancements.
Some of these were st) 11 in process during this inspection.
As a result, only a small sampling of items could be performed in this area.
In general, it appeared that these organizational changes and program enhancements were of benefit to the plant.
However, some weaknesses were noted in the E8TS area.
Or anizational Disci lines and Functions Associated with the EDS The organizational disciplines and functions associated with the EDS appeared to be properly organized and sufficiently staffed.
However, the team noted that some reorganization was in progress during thi s inspection.
The team encourage swift actions to complete recent reorganizational efforts and management actions to minimize any disruptive effects.
The team also considered that the electrical system engineers of the site technical support organization were capable and responsive to operations.
Proactive En ineerin Effort The team reviewed the formation of the component and specialty engineering group under site technical support organization and considered this as a proactive engineering effort.
The assignments of component engineers to address motor operated valves, safety/relief valves, check valves, motors and pumps for various systems and the assignments of predictive maintenance group to address vibration and loose parts monitoring, lubrication oil sampling and analysis for oxidation, corrosion products.and viscosity, thermography, and electrical circuit and insulation testing were examples of good proactive engineering effort.
Personal Understandin of Job Res onsibilit and Duties 5.2 The team noted that under the current site technical support organization, the job responsibilities and interfaces of the component engineers was not well defined to ensure accountability and smooth communication.
There was no written procedure to define the engineering responsibility of component engineers, yet there were procedures defining the responsibilities of the system engineers.
One consequence of this was identified involving the division of. responsibility and accountability concerninq a temporary modification for the installation of a temperature monitoring and data acquisition system on the main steam line to the auxiliary feedwater pump turbine.
In this case, there was confusion noted in implementin~ this temporary modification.
The licensee acknowledged the team s concern and provided a, memorandum dated October ll, 1990, to delineate the responsibilities for the system and component engineers.
This document also defined their interface requirements and inspection findings.
Root Cause Anal sis and Corrective Actions Pro rams The team reviewed the root cause and corrective action programs, and noted that the licensee had initiated a data trending program, This
trending program included data presented in a format to assist licensee system engineers in determining related equipment problems that may warrant root cause investigations.
A positive aspect of the data trending program was that >t attempted to include data on problems which did not cause immediate failures.
The'eam reviewed the data from the first quarter 1990 failure trending report.
The team noted no EDS problems which warranted root cause evaluations beyond those already accomplished by the licensee.
The team also reviewed selected EDS material nonconformance reports (MNCR) and work requests for proper root cause evaluation.
The team noted 'that a reoccurring potential failure mode, cracking of safety battery post seals and covers,'ad no root cause evaluation report.
When questioned by the team, the licensee presented information which showed that the system engineer had performed an evaluation of the cracking as part of a root cause evaluation of battery internal contamination.
The team reviewed root cause evaluation number EER 90-PK-001 which identified that the battery output breaker closed when energized and determined that the problem was due to an incorrect spring location.
However, this evaluation contained no corrective action.
When questioned by the team, the licensee produced records which showed that they had taken action to inspect all other breakers of the same design and found one similar problem.
In both cases, discussed above, the final actions taken by the licensee appeared adequate.
However, the root cause evaluations and corrective actions were not completely documented.
The team noted that the breaker closing and seal cracking problems were initially reported on maintenance nonconformance report (MNCR) forms which were invalidated by the licensee's quality oversight organization.
The licensee reported that these problems were invalidate'd because the problems did not meet the licensee's criteria for an MNCR.
In both cases engineering personnel recognized the validity of the problems and took action to report the problems on different forms.
However, the team noted that no procedure existed to ensure that valid problems, reported on invalidated MNCR forms, were verified to be evaluated and corrected.
As a result of this concern the gA Director and gA Department Director committed to have quality organization personnel ensure that valid problems incorrectly reported on MNCR forms will be correctly documented if the MNCR is invalidated.
The'icensee'eviewed all the MNCRs invalidated in 1990 and concluded that no valid problems had gone uncorrected.
Another weakness the team noted in this area was that in the past, untimely corrective actions had occurred.
For example, problems with the following two LERs were noted.
LER 88-010-01 described an event which resulted in the loss of an auxiliary transformer and consequent reactor trip.
According to this LER, the loss of voltage also resulted in the pressurizer level exceeding TS LCO 3.4.3.1.
The licensee correctly attributed this unanticipated event to a drawing discrepancy.
However, further investiqation revealed that the drawing discrepancy had been identif)ed approximately three years before the LCO was exceeded, but no action had been taken to correct the proble I
LER No. 86-003-00 described an event which resulted in the tripping of Units 1 and 2.
The trips, themselves, were the result of a faulty current transformer (CT).
However, the originating cause was a water leakage from an overhead valve which made )ts way into a load center.
Water was known to be weeping onto the load center, according to the LER.
However, no permanent corrective action had been taken to prevent the water from entering the load center.
With respect to the faulty CT, apparently a history of problems experienced during the construction period prompted the licensee to address the event with the CT manufacturer.
The manufacturer agreed to replace certain CT produced prior to 1980.
However, the licensee did not aggressively pursue the replacement of the CTs until after a
second event involving a similar CT (LER 88-003-00).
. The team reviewed several other EDS root cause evaluations and concluded that those evaluations and corrective actions were adequate.
However, the team co'nsidered that better documentation could enhance these efforts.
In addition, recent LERs reviewed by the team indicated that corrective actions were more timely than those discussed above.
Failure Data Trendin e
5.3 The team reviewed the licensee's component failure trending program.
This program appeared to be quite comprehensive, with good features, such as trending of incipient as well as hard fai lures, trending by component type, manufacturer, etc.
The team, nonetheless, had some concerns regarding the effectiveness of the program.
In particular, a large quantity of records attributed the fai lure to "unknown" causes or to
'ormal aging/ cyclic fatigue."
A corollary of normal aging is that if one component has reached its 'end of service life, other components may have equally reached that plate'au.
Thus the team considered that the licensee may want to evaluate the program to ensure that component failures receive adequate root cause analysis before they are used as input to the trending program.
En ineerin Involvement in Desi n and 0 erations The team reviewed Nuclear Engineering Design (NED) involvement in response to site concerns and.noted that NED engineers were competent and capable.
However, their involvement in site activities was weak in the past.
This was. previously recognized, and the licensee has undertaken efforts to reorganize and enhance efforts in the engineering area.
The team found that their involvement with and response to the site has been improving as evidenced by a number of items such as system status reports which documented recent NED engineer system walkdowns.
In addition, the team noted that the NED electrical design supervisory staff was supportive of site concerns.
Since the establishment of Site NED, an approximate 75K reduction in turnaround time was.observed in dealing with issues.
En ineerin Su ort and In ut to Site Testin and Maintenance The team reviewed a number of instruction change requests on diesel generator maintenance for evidence of adequate engineering support and
0 t
5.4 input.
The documents reviewed showed active electrical and mechanical system engineering participation from Site Technical Support Organization.
Coordination of Interfaces of En ineerin Amon Various Disci lines The coordination of site personnel with NED design engineers was historically through system engineers from the site technical support organization and continued this way after the establishment of the site NED.
The team found that maintenance standards personnel contacted design engineerinq through system engineers and at the same time kept system engineers conformed.
Electrical system engineers of the site technical support group were responsive to operations.
However, operations personnel indicated that the responsiveness of the system engineers was somewhat affected by a large turnover of personnel and the new system engineers were weak in their knowledge of their respective systems for a period of time.
Timel Closeout and Backlo of En ineerin Items For the week of October 15, 1990, the backlog of EER items for the Site Technical Support organization was 2343 open, 50 new, 49 transferred,
returned, 9 on hold, and 78 closed.
For the same period, the Electrical Systems Group had the backlog of 243 open with 136 in-house and 107 transferred.
The team noted the trend of the backlog has been decreasing for the last three months.
It was considered that the number of open EERs was just an indication and could not be used in the evaluation of the performance of the electrical systems group without giving due consideration to the complexity of each EER.
The team reviewed NED's performance on resolving EERs and found 4 open EERs two years old and 33 open EERs one year old.
The average days to close an EER in July was 139 days and the target was 90 days.
Self-Assessment and Trainin Self Assessment The team reviewed the licensee self-assessment program and considered it generally to be adequate.
The following observations were made:
Organization The self-assessments were performed by a number of different groups.
The team noted that organizations such as Nuclear Safety Department, equality Assurance, Independent Safety Engineering, Corporate Assessment Group, Nuclear Engineering Assurance, Plant Review Board, Design Review Board, Off-site Safety Review Committee, Plant Modification Committee, Human Performance Evaluation, and Incident Investigation Teams performed assessment Independence The team noted that most of the aforementioned self-assessments were performed by the responsible organization and management being assessed.
Coordination The team was informed that regular meetings were scheduled,to coordinate the various self-assessment groups.
- Due to the complexity of the organization, the team viewed that the coordination effort could be a problem.
The team found that special projects initiated under the self-assessment program were proactive and generally adequate.
Some special projects were as follows:
'iesel Generator Safety System Functional Inspection (Summer, 1990)
Electrical Distribution Assessment (Spring 1990)
The team considered these efforts commendable, but was concerned that they did not identify the significant discrepancies noted during this inspection (e. g., calculation problems in not considering the worst case loading conditions).
As a result, the team considered that current efforts toward the design basis reconstitution should be more detailed and ensure the original design basis assumptions are challenged.
The team considered the following example commendable with regard to the licensee's effort in identifying deficiencies and weaknesses.
equality deficiency report ((DR) 90-0356 was issued on September 13, 1990, for errors in an original design calculation, that were translated into plant maintenance tasks and SIMS data, resulting in erroneous level indication for the diesel fuel oil storage tank.
On September 10, 1990, a quality engineer discovered that the incor rect result of design calculation 13-JC-DF-A07 was used by maintenance personnel during the calibration of diesel fuel oil storage tank level instrumentation 13-J-DFN-LI-0033/0034.
The use of this calculation output caused the indicated tank ldvel at the diesel generator control board to read greater than the actual level in the normal operating range which could have caused inadvertent entry into the a'ction statements for TS 3.8. 1. 1 or 3.8. 1.2.
The calibration error was that the OX - lOOX indicated range was an actual tank level of 1.5" - 150.5" or 1X - 96.5X.
The team considered this a significant recent finding and encouraged the continued support of the licensee management in the self-assessment area.
Mana ement Observation Pro ram The team noted that the Management Observation Program was initiated to provide guidance for management field observations.
Site management was assigned a plant walkdown once a month with a written report documenting the walkdown.
In addition, offsite directors/managers were assigned a
plant walkdown once a quarter.
The team considered this as a program good for promoting management involvement with plant activities.
The
0 6.0 team encouraged continuation of this effort and also considered that the licensee might gain added benefit by performing these walkdowns more often.
Em lo ee Concerns Pro ram The team found the employee concerns program was a means for personnel to elevate their problems above their direct supervisor in a confidential manner and to ensure the freedom of individuals to report problem.
The team observed an overall downward trend in the number of concerns since June 1989.
One electrical item was noted among the recent concerns.
The timeliness of closeout was 26.6X of the items in less than 30 days, 16.9X, in 30 to 60 days, 19.5X in 60 to 90 days, and 37X greater than 90 days.
~Trainin The team considered that onsite engineering personnel appeared to be adequately trained to perform their jobs.
The results of the new, training programs instituted were not observed, but the formulated training program appeared to be adequate.
The licensee indicated that the goal was to establish 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> of training for each system engineer per year.
A position of Site Engineering Training Coordinator was established to lan, schedule, and record training of site engineers and technicians.
he program was still in development and not fully implemented, and the results of the training program were not observed by the team.
The team found that the training program was not affected by the ongoing reorganization, and the continuing support from the management was indicated by the resources allocated.
The team also noted Onsite Nuclear Training Group was not affected by the ongoing reorganization and had its full supplement of the budget.
The group was staffed with approximately 135 individuals.
Within the 135, approximately 100 were APS employees.
The licensee indicated that their intention was to have the group be staffed by APS employees and to retain the experience in-house.
The training program included certified INPO programs, training records and management, training program description, job qualification standards, procedures training, and maintenance training for atmospheric dump valves and motor operated valves.
The team also noted that the Corporate Nuclear Engineering Program
Training was being abolished during the ongoing reorganization and was to be merged with Site Nuclear Engineering and Construction Organization training.
Within this area inspected, no violations or deviations were identified.
GENERAL CONCLUSIONS As a result of the problems noted in the engineering area, the team considered that the electrical design calculations have been neglected and that they were either missing, inadequate, or lacked sufficient review to demonstrate the adequacy of the EDS design to'eet GDC 17 requirements under worst case conditions.
That GDC 17 was met was then
demonstrated by APS as a result of this inspection, with the exception of supplying power to three units from two startup transformers.
As a result, administrative controls needed to be implemented for the startup transformers under those conditions.
The team considered that APS personnel were responsive to the team's concerns and follow-up efforts were commendable, The team noted that a
design basis reconstitution effort is in progress and that the licensee intends that calculation problems such as those noted above will be corrected during this effort.
The team encouraged continuation of these efforts.
In addition, the team considered that it may be appropriate to look at additional scenarios to ensure that other problems do not exist such as operation of three units on two startup transformers.
Also, the team encouraged APS to ensure that the computer code being used to determine the adequacy of the EDS be validated.
The team noted that the design engineerinq training program has not been fully implemented and that the design engineers have not been extremely involved with the plant in the past.
These coupled with the subtleties of the EDS may be sufficient for APS to evaluate whether there may be a
need for some adjustments to these programs before implementation of the EDS design basis reconstitution effort.
The. team further concluded that there was a disconnect between the site and nuclear engineering in the past based upon the spray pond modification.
APS may want to ensure that there is a proper feedback mechanism for assurance that the engineering program upgrades are doing what was originally intended. "
The team was also concerned that APS has completed an SSFI, an electrical systems reliability study, and an electrical system assessment on the EDS and none of these efforts, identified concerns with calculations, as noted above.
Thus, the team considered that the current efforts should be more detailed.,
Mith regard to maintenance and surveillance issues, the team considered that the findings'ndicated a weaknesses in preventive maintenance and testing programs.
Previously, APS performed a number of root cause evaluations for events such as transformer problems, an electrical systems reliability study, and an electrical systems assessment.
In these reviews, APS noted that there were problems in high voltage equipment that could be attributed to maintenance/testing deficiencies.
The licensee appeared to have satisfactorily resolved these particular issues and started to implement a
PM task force to address weaknesses in this area.
However, the team noted that some of the items reviewed by the team were not being considered for inclusion into any PM or surveillance program, and that the RCM program had been suspended.
Thus,
~
o
~
the team considered that APS may want to continue to challenge the assumptions made in preparation of the PM effort.
7.0.
EXIT MEETING The inspector met with licensee management representatives periodically during the inspection and held an exit meeting on November 9, 1990 to discuss the areas reviewed during the inspection, weaknesses observed and findings.
The licensee did not identify any documents or processes as proprietary, and acknowledged the team's finding Si nificant Documents Reviewed ATTACHMENT A Document Number 13-EN-300 32MT-9ZZ74 32MT-9ZZ31 30DP-OAP01 32MT-9ZZ58 73DP-OAP01 32ST-9ZZ74 32ST-9PK02 Title Installation Specification for Electrical Cables in Cable Trays Appendix "R" Molded Case Circuit Breaker Test Motor Control Center Molded Case Circuit Breaker and Cubicle Test Maintenance Instruction
'Writer's Guide
. Maintenance of Inverters Writer's Handbook for Surveillance Test Procedures Molded Case Circuit Breaker Surveillance Test 92-Day Surveillance Test of Station Batteries Revision/
Procedure Change Notice PCN)
2/PCN
2/PCN
1/PCN
1/PCN
0/PCN 1 3/PCN 1 Date 08/28/87 05/23/90, 01/05/90 10/04/90 09/23/89 07/21/90 01/18/90 06/Ol/90 32ST-9PK01 32ST-9PK03 13-EN-301 7-Day Surveillance Test of Station Batteries 18-Month Survei 1 lance Test of'Station Batteries Installation Specification for Electrical Cables in Conduit and Duck Bank 5/PCN 1
,
06/07/90 06/01/90 0/SCN 2 05/16/90 (Specification Change Notice)
0 Cal cul ati ons Number 13-EC-MA-202 13-EC-MA-212 13-EC-MA-610 13-EC-NA-220 13-EC-NA-301 13-EC-PA-210 13-EC-PB-100
. 13-EC-PB-101 13-EC-PB-110 13-EC-PE-100
'13-EC-PG-100 13-EC-PG-220 13-EC-PH-240 13-EC-ZA-300 (Not numbered)
13-MC-DF-306 Procedures Number Revision Title
7
1 Revision Title System Constants Auxiliary System and Transformer Sizes Voltage Regulation Study EPA Overload and Fault Current Protection - 13.8 kV Startup Power Transformer Protection Power Cable Ampacities Class lE Metalclad Switchgear 4. 16 KV-Protection Undervoltage Protection Study Protection Coordination Study for Safe Shutdown Power and Control Circuits Diesel Generator Instrument Transformer Ratio and Accuracy Selection Class lE 400 Y, 750 kVA Load Center Protection Electrical Penetration Assemblies Backup 08/22/86 Protection, 480 V LC Loads Electrical Penetration Assembly Backup Protection, 480 V MCC 8 Miscellaneous Loads Derating of Cables in Trays Palo Verde Nuclear Generating Station Electrical Load Evaluation As-Built Calculation For Diesel Fuel Storage and Diesel Generator Day Tanks Date 07/12/84 ll/16/84 08/02/90 07/26/84 08/20/84 07/12/84 04/15/85 04/23/84 04/29/85 07/03/90 04/15/85 06/29/90 05/15/85 11/02/90 09/25/90 Date 7N431.02.01 73AC-ODC01 81AC-OEE01 430P-3NA01 S stem Descri tions
1 Protection, Metering 8 Automated Control 03/03/90 Instructions for Palo Verde Protective Device Settings Relay Setting Sheet Control'rotective Relaying and Coordination Procedure Palo Verde Nuclear Generating Station Manual, 13.8 kV Electrical System (NA)
09/19/88 ll/22/88 10/10/86 Number Revi si on Title PB
PG
PH
Class Class Class Class 1E 4.16 kV Power System lE Standby Generation System lE 480 V Power Switchgear System 1E 480 V Power MCC System Date 09/23/86 10/28/87 09/23/86 10/28/87
T ( 5
e
~Licensin Number Miscellaneous Revision Title
Chapter 8 - Electric Power, PVNGS Updated FSAR Section 3/4.8 - Electrical Power Systems, Technical Specifications, Palo Verde Nuclear Generating Station, Unit 1 Date 03/20/90 03/22/88 Number Revision Title
'(Not numbered)
EER89-XE-060 LTR. 167-00428-JTB/MB PCN 12 Section 4, Electrical General Design Criteria, Design Criteria Manual, Part II Motor Operated Valves Response to Bechtel Power Corporation's Comments Regarding AC Load Test Procedure (Including Attachments)
Procedure Change Notice, Procedure No. 430P-3NA01, 13.8 kV Electrical Sy'tem (NA)
Date 10/09/87 12/05/89 04/29/87 10/26/90
'4 ~