IR 05000528/1988044
| ML17304B047 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 02/16/1989 |
| From: | Coe D, Fiorelli G, Miller L, Polich T NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION V) |
| To: | |
| Shared Package | |
| ML17304B046 | List: |
| References | |
| 50-528-88-44, 50-529-88-42, 50-530-88-41, NUDOCS 8903060158 | |
| Download: ML17304B047 (40) | |
Text
U.
S.
NUCLEAR REGULATORY COMMISSION
REGION V
Re ort Nos.
Docket Nos.
License Nos.
Licensee:
50-528/88-44, 50-529/88-42 and 50-530/88-41 50-528, 50-529, 50-530 NPF-41, NPF-51, NPF-74 Arizona Nuclear Power Project P.
0.
Box 52034 Phoenix, AZ. 85072-2034 Ins ection Condu ed:
Dece e
Inspectors:
P lich, S
Resi ns Fi6r li, ent Inspector Approved By:
L
. Miller Chief Reactor ProjectsSection II Ins ection Summar
1988 through January 27, 1989 2-l~
-h'e i>dent Inspector Date Signed 2.-is-P ctor Date Signed lf. -g~
Date Signed
Date Signed Ins ection on December
1988 throu h Januar
1989 Re ort Nos.
50-528/88-44 50-529/88-42 and 50-530/88-41 Areas Ins ected:
Routine, onsite, regular and backshift inspection by the three resident inspectors.
Areas inspected included: previously identified items; review of plant activities; engineered safety feature system walkdowns; monthly surveill'ance testing; monthly plant maintenance; over greasing of motor and generator set - Unit 1; engineered safety features (ESF) relay test error - Unit 1; slipped control element assembly (CEA) No.
52 - Unit 1; loss of both engineered safety features (ESF) service transformers at Palo Verde Nuclear Generating Station - Unit 2; damaged battery cell - Unit 2; reactor power cutback - Unit 2; 3A emergency diesel generator (EDG) failure - Unit 3; entry into Mode 5 (cold shutdown) - Unit 3; fai lure to follow procedures
- Unit 3; charging pump block cracks
- Units 1 and 3; dropped part length control element assemblies (CEA's) - Unit 3; review of licensee event reports
- Units 1, 2 and 3; and review of periodic and special reports-Units 1, 2 and 3.
8903060158 890216 PDR ADOC)( Q5QQQ528
r n>u
During this inspection the following Inspection Procedures were utilized:
30702, 60705, 62703, 71707, 71710, 90712, 92700, 92701, 92702, 93702.
Safet Issues Mana ement S stem (SIMS) Items:
None Results:
Of the 18 areas inspected, no violations were identified.
General Conclusions and S ecific Findin s
Strenghts noted were:
Well coordinated and timely response to an Emergency Diesel Generator rocker arm and turbocharger fai lure.in Unit 3 (see Section 13).
Weaknesses noted were:
Procedural weakness with maintenance and chemistry procedures (see Sections 7 and 14).
Non adherance to procedures (see Section 8).
Si nificant Safet Matters:
None Summar of Violations:
Summar of Deviations:
0 en Items Summar None None 5 items closed, No items left open, and
new items opene DETAILS 1.
Persons Contacted:
The below listed those contacted:
technical and supervisory personnel were among Arizona Nuclear Power Pro ect ANPP
"R.
- J P.
"R.
C.
L.
J.
W.
W.
R.
D.
"R.
AJ
)hW D.
AJ J.
A.
R.
R.
J.
K.
A.
D.
AJ Adney, Allen, Brandjes, Buckingham, Butler, Churchman, Clyde, Dennis, Doyl e, Fer now, Ferro, Fowler, Gouge, Haynes, Ide, Karner, Kirby, LoCicero, McCabe, McKinney, Middleton, Minnicks, Oberdorf, Ogurek, Papworth, Phi 1 1 ips, Scott, Shriver, Si 1 1 s, Sowers, Stover, Tench, Younger, Zeringue, Assistant Plant Manager, Unit 2 Plant Manager, Unit 1
2 Manager, Central Maintenance Operations Manager, Unit 2 Director, Standards and'echnical Support Manager, Work Control, Unit 3 Supervisor, Shift Technical Advisors Manager, Work Control, Unit 1 Consultant, Site Services Manager, Training Manager, Chemistry, Unit 2 Manager, guality Systems and Engineering Operations Manager, Unit 3 Vice President, Nuclear Production/Site Director Plant Manager, Unit 2 Executive Vice President, ANPP Administration Director, Nuclear Production Support Manager, Independent Safety Engineering Maintenance Manager, Unit 1 Operations Supervisor, Unit 1 Operations Supervisor, Unit 2 Maintenance Manager, Unit 3 Manager, Radiation Protection, Unit 1 Manager, Radiation Protection, Unit 2 Director, guality Assurance Manager, Maintenance, Unit 2 Manager, Work Control, Unit 2 Manager, Compliance Supervisor, Radiation Protection Standards Manager, Engineering Evaluations Acting Manager, Nuclear Safety Director, Site Services/Manager, Material Control Manager, Plant Standards and Control Plant Manager, Unit 3 The inspectors also talked with other licensee and contractor personnel during the course of the inspection.
~Attended the Exit meeting held with NRC Resident Inspectors on February 1,
198.
Prey ious 1 Identi fi ed Items - Units 1
and
92702 92701)
(Closed Followu Item 528/87-39-01:
"Ensure Reviews to Fire Protection Procedures Included Review of All Maintenance Work Orders" - Unit 1.
This item is related to a problem involving the restoration of fire barriers following maintenance work.
The licensee informed the NRC in a letter dated March 10, 1988, that maintenance guidance provided to the maintenance planners/co-ordinators would be reviewed and modified if appropriate.
The inspector confirmed that the Work Control Procedure 30AC-9ZZOl was revised to include an instruction requiring compliance with fire barrier restoration provisions in procedure 14AC-9FP02
"Fire Barrier Seal and Structural Steel Fireproof Removal and Reinstallation.
This item is closed.
Closed Followu Item 528/88-01-24
- "0 eratin Ex erience/
Pro ram Evaluation" - Unit l.
This matter dealt with an observation related to the long period of time it took responses to be received from responsible organizations in connection with commitment tracking items such as NRC Information Notices.
In the past, the responsible departments were allowed to change and extend their due dates for items on the Regulatory Commitment Tracking System (RCTS).
This can no longer be done.
The responsible department must request an extension through the appropriate licensing engineer.
A one month extension may be granted as requested by the department supervisor or manager, and anything beyond a month requires a
memo from the department to the licensing manager justifying an extension.
Procedure 93GB-OLC06, "Regulatory Tracking Procedure" will be revised to state this change.
In addition, a biweekly report is sent to management identifying the number of open/overdue items.
Supervisors and department RCTS coordinators also receive the open/overdue RCTS commitment list for resolution every two weeks.
These actions should improve the timeliness of responses associated with NRC Information Bulletin followup actions.
This item is closed.
(Closed)
Enforcement Item (528/88-24-01:
"Essential Chilled Water S stem Ino erable for Nine Da s in Mode 1" - Unit 1 On May 29, 1988, with Unit 1 operating at 100K, the licensee discovered that the root isolation valves for a flow transmitter which supplied a compressor
"run" permissive were shut.
This condition which was found on both trains of essential chilled water system existed for nine days.
The cause of the problem was determined to be personnel error and
was related to the closure of the wrong isolation valves during an attempt to separate a "non-class" instrument which was attached to a class system.
Also, involved in the err'or was the posting, in the wrong location, of plastic tags with instructions to maintain the non-class instrument valves closed.
The inspector verified the tags were now properly posted and confirmed that quarterly surveillance tests had been conducted on both trains of essential chillers since the event.
The
licensee's corrective actions included the development of a procedure dealing with the control of warning tags.
This procedure is currently in the review and approval phase.
Other longer term corrective actions related to communications and auxiliary operator training were also identified by the licensee.
Escalated Enforcement Action No.88-182 dated December 1,
1988 was issued to the licensee for the closure of the wrong valves which made both trains of essential chilled water system inoperable.
The licensee's response to the enforcement letter was reviewed by the NRC and found adequate.
Closed Enforcement Item (528/88-07-02
"Review of Modifi-cation Inade uate" - Unit 2.
This item resulted from rendering a steam driven auxiliary feedwater pump inoperable due, in part, to an inadequate review of modifications made to system components.
As part of the licensee's commitment for corrective action, the guality Systems and Engineering audit of Engineering Evaluation Requests was forwarded to the Nuclear Engineering Department (NED) for final review.
This review was completed with no further comment, completing the corrective action commitment for this item.
The inspector reviewed the memorandum documenting the NED review.
Other corrective actions were reported in NRC inspection report 528/88-32.
This item is closed.
3.
Revi ew of Plant Activities (71707 71710 93702 Unit 1 Except for brief periods when special surveillance tests were being conducted at reduced levels the plant operated at essentially 100%. throughout the period.
Unit 2 Unit 2 operated at 100% power from the start of the report period until December 22, 1989, when a reactor power cutback occurred.
The cause of the cutback was identified (See Section
12) and the Unit was returned to service on December 23, 1988.
The Unit remained at 100K power for most of the report period.
c.
Unit 3 Unit 3 operated at lOOX power from the start of the inspection period until January 6, 1989, when the unit was shutdown in accordance with Technical Specifications due to an inoperable emergency diesel generator (see paragraph 13).
It remained shutdown until January 22 and returned to 100K power operation on January 23, 1989.
Power operation at lOOX continued through the end of the report period with the exception of brief power reductions on January 26 and 27th following two separate dropped rod events (see<paragraph 17).
Plant Tours The following plant areas at Units 1, 2 and 3 were toured by the inspector during the course of the inspection:
0'0
0
0
0 Auxi 1 iary Bui 1 ding Containment Building Control Complex Building Diesel Generator Building Radwaste Building Technical Support Center Turbine Building Yard Area and Perimeter The following areas were observed during the tours:
1.
0 eratin Lo s and Records Records were reviewed against Technical Specification and administrative control pro-cedure requirements.
The inspector noted that following the return of Unit 3 to power operation on January 22, 1989, the unidentified Reactor Coolant System (RCS) leak rate taken on January
was 0.692 gpm, which was significantly larger than the approximately 0.3 gpm rate prior to the maintenance outage.
Through a series of leak rate tests, equipment inspections, and containment entries, the licensee determined the source of additional unidentified leakage to be a leaking check valve in the Chemical and Volume Control System discharge header.
The leaking flange of this valve (3CHN-V435) was sealed with an injectable sealant compound during the maintenance outage, but the condition was not remedied.
The total and unidentified leak rates were within Technical Specification limits, and at the end of the reporting period the licensee had not decided if further action would be taken.
The inspector noted that the licensee quantified the leak rate from valve 435 to be approximately 0.4 gpm and that
i
/
total and unidentified rates remained stable.
The licensee planned to continue monitoring leak rates for upward trends, and to repair valve 435 during the refueling outage.
These actions appeared adequate.
Monitorin Instrumentation Process instruments were observed for correlation between channels and for conformance with Technical Specification requirements.
3.
observed for conformance with 10 CFR 50.54.(k), Technical Specifications, and administrative procedures.
4.
E ui ment Lineu s
Various valves and electrical breakers were verified to be in the position or condition required by Techni cal Speci ficati ons and admini strati ve procedures for the applicable plant mode.
This, verification included routine control board indication reviews and the conduct of partial system lineups.
5.
E ui ment Ta in Selected equipment, for which tagging requests had been initiated, was observed to verify that tags were in place and the equipment was in the condition specified.
6.
General Plant E ui ment Conditions Plant equipment was observed for indications of system leakage, improper lubrication, or other conditions that would prevent the systems from fulfillingtheir functional requirements.
Unit l auxiliary feedwater pumps had oil and water leaks that were not identified by maintenance required tags.
The control room operators were unaware of any problems with the pumps when questioned by NRC inspectors and management.
This situation was brought to the attention of Unit 1 management who stated the Independent Safety Engineering Group (ISEG) had made the same observations of the pumps during a recent report.
The licensee later informed the inspector that some of the observed leaks had been repaired.
A reinspection of the area approximately 2 weeks later disclosed that some standing oil still existed on the "A" auxiliary feedwater pump base.
The licensee stated these leaks (which were minor in nature)
would require removing the pump from service.
The inspector did not consider that the leaks affected pump operability.
The inspector also noted that the "8" auxiliary pump seals were leaking.
This condition likewise does not affect pump operability.
These repairs also would require removing the pump from service.
Followup on the licensee corrective actions will be pursued as part of the routine inspection progra.
Fire Protection Fire fighting equipment and controls were observed for conformance with Technical Specifications and administrative procedures.
8.
Plant Chemistr Chemical analysis results were reviewed for conformance with Technical Specifications and admin-istrative control procedures.
Procedure 74AC-9CYY04, Revision 0, "System Chemistry Specifications" was noted to be confusing with regard to hydrogen specifications when entering Mode 5 and securing reactor coolant pumps (see Section 14).
9.
~Securit Activities observed for conformance with regulatory requirements, implementation of the site security plan, and administrative procedures included vehicle and personnel access, and protected and vital area integrity.
10.
Plant Housekee in Plant conditions and material/equipment storage were observed to determine the general state of cleanliness and housekeeping.
.Housekeeping in the radiologically controlled areas was evaluated with respect to controlling the spread of surface and airborne contamination.
ll.
Radiation Protection Controls Areas observed included control point operation, records of licensee's surveys within the radiological controlled areas, posting of radiation and high radiation areas, compliance with Radiation Exposure Permits, personnel monitoring devices being properly worn, and personnel frisking practices.
No violations of NRC requirements or deviations were identified.
4.
En ineered Safet Feature S stem Walkdowns - Units 1 2 and
71710 Selected engineered safety feature systems (and systems important to safety)
were walked down by the inspector to confirm that the systems were aligned in accordance with plant procedures.
Ouring the walkdown of the systems, items such as hangers, supports, electrical cabinets and cables, were inspected to determine that they were operable, and in a condition to perform their required functions.
b.
Accessible portions of the following systems were walked down during this inspection period.
Unit 1.
o
"A" Train Low Pressure Safety Injection System o
"A" Train High Pressure Safety Injection System o
"A" and "B" Trains Essential Cooling System
Unit 2 o
"A" Train Low Pressure Safety Injection System o
"A" Train High Pressure Safety Injection System o
Channels
"8" and "D", 125V DC Class lE Battery System Unit 3 o
"A" Train Low Pressure Safety Injection System o
"A" Train High Pressure Safety Injection System o
"A" and "B" Trains Essential Cooling System o
"A" Train Emergency Diesel Generator Miscellaneous Accessible portions of the fol:lowing systems were walked down during this inspection period.
o Plant Fire Protection Pumping System No violations of NRC requirements or deviations were identified.
5.
Monthl Surveillance Testin
- Units 1
. 2 and
61726 Surveillance tests required to be performed by the Technical Specifications (TS) were reviewed on a sampling basis to verify that:
1) the surveillance tests were correctly included on the facility schedule; 2)
a technically adequate procedure existed for performance of the surveillance tests; 3) the surveillance tests had been performed at the frequency specified in the TS; and 4) test results satisfied acceptance criteria or were properly dispositioned.
b.
Portions of the following survei llances were observed by the inspector during this inspection period:
Unit 1 Procedure Descri tion o 36ST-9SB02 Plant Protection System Bistable Trip Units Functional Test "D" Channel o 41ST-1ZZ23 Control Element Assembly Position Data Log o N/A Reactor Coolant Pump Vibration Measurements Unit 2 o 36ST-2SE02 Excore Linear Monthly Calibration "C" Channel
Unit 3 o 36ST-2SE02 Shutdown Margin No violations of NRC requirements or deviations were identified.
6.
Monthl Plant Maintenance
- Units 1 2 and
62703 During the inspection period, the inspector observed and reviewed documentation associated with maintenance and problem investigation activities to verify compliance with regulatory requirements, compliance with administrative and maintenance procedures, required gA/gC involvement, proper use of safety tags, proper equipment alignment and use of jumpers, personnel qualifications, and proper retesting.
The inspector verified that reportabi lity for these activities was correct.
The inspector witnessed portions of the following maintenance activities:
Unit l Descr'i tion Calibration of "A" Train Essential Chiller Cooler Pressure Gage.
Calibration of "8" Train Essential Chi lier s Low Refrigerant Temperature Protection Switch.
Troubleshooting of "B" Train Emergency Diesel Generator Essential Room Exhauster Breaker Trip.
Removal of "B" Charging Pump.
Unit 2 Descri tion o
Calibration of "A" Train Emergency Diesel Fuel Transfer Flow Instrument.
o Installation of Bushings, in Engineered Safety Features (ESF) Transformers NBN-X03 and NBN-X04.
o Replacement of "B" Train Engineered Safety Features Electrical Relay K-302 (Steam Generator Blowdown Isolation).
Unit 3 Descri tion o
"A" Train Emergency Diesel Generator Rocker Arm and Turbocharger Replacement.
o
"A" Train Low Pressure Safety Injection Pump Suction Valve Repairs.
No violations or deviations of NRC requirements were identified.
7.
Over reasin of Motor-Generator Set - Unit 1 62703
.On January 21, 1989, the bearings on a Control Element Drive Mechanism Motor-Generator were over greased.
This resulted from conflicting instructions provided in the work package.
One instruction stated that a maximum of 3-6 ounces of Shell Dolium-R grease should be used.
A portion of the technical manual attached to the work package provided an instruction which stated to attach a
grease gun at each input grease fitting and fill with grease until grease comes out the drain tubes.
The mechanic followed the latter instruction and inserted approximately 12 ounces of grease in one bearing and approximately 16 ounces in another.
After realizing that the amount of grease inserted appeared in excess, the mechanic reported this condition to the maintenance work control coordinator.
The motor was inspected and neggered and no problems were encountered.
The motor generator set which is non-safety related was eventually declared operable and phased into operation.
The licensee is currently reviewing the matter.
Corrective actions will include a change to the technical manual.
Followup action will be pursued as part of the normal inspection program.
The inspector concluded that the licensee has properly identified and corrected this error.
No violations or deviations of NRC requirements were identified.
8.
En ineered Safet Features ESF) Rela Test Error Unit 1 61726)
On January 6, 1989, during the performance of surveillance test 36ST-9SA98 "Engineered Safety Features Actuation System Train "B" Subgroup Relay 15 Day Functional Test", the instrument and control (I/C) test performers actuated the wrong ESF relay.
Instead of actuating auxiliary feedwater actuation system
"B" Train relay K-413 was in ESF auxiliary relay cabinet 1JSABC01,
"A" Train relay K-413 actuated.
This occurred because the technicians entered the wrong relay cabinet 1JSAAC01.
All test switch labeling is identical in both cabinets, only the cabinet labels are different.
As a result of this error, No.
1 Steam Generator blowndown isolation valve UV-500/, and two steam generator sample line isolation valves closed instead of the corresponding No.
2 steam generator valves.
The inspector confirmed that no plant problems occurred as a result of the error.
Entry into the wrong cabinet was contrary to procedural
instructions., Alert control room operators observed that the wrong component indicating lights came on and immediately halted the testing.
The I/C personnel were disciplined for their actions by Unit 1 management.
The licensee's event investigation was timely and corrective actions appeared adequate to prevent reoccurrence.
No. violations or deviations of NRC requirements were identified.
9.
Sl i ed Contr ol Element Assembl (CEA No.
52 - Unit 1 71707 On December 21, Control Element Assembly (CEA) No.
52 slipped into the core when control room operators were moving the CEA for axial shape index control.
The part length CEA slipped approximately 25 inches and could not be withdrawn.
Actions required by the facility Technical Specifications were'implemented, one of which involved a power reduction to 80%.
Following the troubleshooting by the system engineer, it was determined that the problem was caused by a bad logic card.
Replacement of the card was successful in allowing CEA No.
52 to be withdrawn from the core and aligned with the other CEAs in its subgroup.
The inspector reviewed the licensee's actions associated with the slipped CEA and determined that the required Technical Specification requirements had been met.
No violations of NRC requirements or deviations were identified.
10.
Loss of Both En ineered Safet Features (ESF) Service Transformers at Palo Verde Nuclear Generatin Station - Unit 2 71707 Palo Verde Nuclear Generating Station Unit 2 declared an Unusual Event at 7:59PM, January 3, 1989, when power from both engineered safety features (ESF) service transformers 2E-NBN-X03 and 2E-NBN-X04 was interrupted due to lightning strikes on each of the lines supplying power to the transformers.
Both emergency diesel generators started up on loss of the transformers as designed.
The plant was operating at 100% power at the time of the incident.
With both sources of offsite power lost to the ESF busses, the licensee entered Technical Specifications 3.8.8. 1. 1.a.
The Unusual Event was terminated at 4:49 p.m., January 4, 1989, after one source of offsite power was restored to one of the ESF busses.
The second source of offsite power was restored at 7:25 p.m., January 4,
1989, at which time the Technical Specification action statement was exited.
Repairs to the two ESF service transformers included the replacement of two bushings on each transformer.
Tests of relays, breakers, oil and windings associated with the transformers confirmed no other damage had resulted from the lightning strikes.
The inspector confirmed that the licensee had performed the proper surveillances and entered the proper action statements required by the Technical Specifications.
The inspector also observed a portion of the installation of the bushings and surveillance test of the
emergency diesel generators following restoration of power to the ESF busses.
While observing the initial installation of the bushings, the inspector was made aware that the replacement bushings obtained from the warehouse were 21 inches long while the damaged bushings were
inches long.
Twenty-one inch bushings were also installed in the spare transformer stored onsite.
Both size bushings were identified with the same catalog number.
Replacement bushings of identical size to the damaged bushings were eventually located at another ANPP storage location and installed.
The significance of having the same catalog number for both sized bushings will be pursued with the licensee (Inspector fol lowup Item 529/88-42-01).
No violations of NRC requirements or deviations were identified.
ll.
Dama ed Batter Cell '- Unit 2 (71707 92700 On January 16, 1989, at 1808 licensee electricians performing a
surveillance test on the Unit 2 "D" battery found a hole in the top of the battery jar which contains cells 19 and 20.
The hole was on the, cell 20 side near the battery terminals..
The licensee was not able to determine the exact time of the damage and as a result initiated an event investigation into the damage.
(Inspector Followup Item 529/88-42-02)
When the problem was reported to the control room'he licensee entered Technical Specification (TS) Limiting Condition for Operation (LCO) 3.8.2. l.a.
The LCO required restoring the DC Train within 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> or be in Hot Standby within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />, and Cold Shutdown within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.
The battery breaker was opened to jumper out cell 20 at 2141 and closed at 2211 after the cell was jumpered out.
Although cell 20 was jumpered out the Engineering Evaluation of the lower voltage had not been completed.
At 2233 a power reduction was initiated to comply with Technical Specification 3.8.2. 1.
The NRC Duty Officer and Resident Inspector were notified of the power reduction.
Temporary modifi.cation 2-89-PK-001 was written to jumper out cell 20.
Subsequently Engineering Evaluation Request EER-89-PK-002 verified that the battery was operable with cell 20 jumpered out.
At 2356 TS 3.8.2. l.a was exited and the power increase was began from 71%.
The unit was returned to 100% power at 0400 on January 17, 1989.
12.
Reactor Power Cutback - Unit 2 93702 On December 22, 1988, at 1440 while operating at 100% power Unit 2 experienced a turbine trip and a subsequent reactor power cutback to approximately 45% power.
Operators stabilized the plant and further reduced power to 35%.
The inspector was in the control room at the time of the turbine trip and monitored control room response.
The licensee's investigation into the event determined the cause of the event to be an inadvertent isolation of control air from the normal
level dump valves for the moisture separator
"C" drains to the heater drain tank.
The high level dump valve was mechanically isolated from the condenser for maintenance causing moisture separator drain tank level to increase and initiate a high moisture separator level trip of the turbine.
The licensee further determined the control air supply hose to the high and normal level dump valves had been reversed.
The attempted isolation of the high level dump control air for maintenance actually isolated the normal level dump valve control air which failed closed and initiating the event.
(Inspector Followup Item 529/88-42-03)
13.
3A Emer enc Diesel Generator EDG Failure - Unit 3 62703 On January 3, 1989, the Unit 3 "A", EDG was taken out of service to perform planned maintenance.
On January 4, while performing a four hour engine analysis, EDG "A" tripped on "overspeed" caused by excessive vibration near the overspeed switch after running approximately three hours.
The licensee determined the 8L cylinder rocker arm had broken and, after obtaining the part from the vendor, made repairs and started EDG "A" for post maintenance testing.
During post maintenance testing a second high vibration trip occurred on January 5.
After consultation with the vendor, EDG "A" was once again started and loaded to 3.0 MW.
After hearing a
harmonic noise at 3.0 MW the licensee decided to shutdown the diesel and commence a reactor power reduction to comply with 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> Technical Specification action statement 3.9. l. 1.b, (AC Sources).
Before the diesel unloading began, a third high vibration trip occurred.
At 0140 on January 6,
a Unit 3 power reduction began and the unit entered Mode 5 (cold shutdown) at 1032 on January 7,
1989.
Investigation showed severe damage to the turbocharger compressor blading.
With the vendor's assistance, the licensee determined the turbocharger damage was caused by resonance of the blades during back pressure surges from cylinder 8L (exhaust),
following failure of the rocker arm.
The fai lure of the rocker arm was determined by the licensee to be the result of fatigue fai lure of a defect created when the rocker arm was manufactured.
The licensee performed visual inspections on all remaining EDG rocker arms in all three units.
One additional exhaust rocker arm on cylinder 9R was found to have a small crack in approximately the same location as the failed rocker arm and was found on the same EDG (3A) as the failed rocker arm.
The licensee procured new quality-related turbocharger parts and commercial grade rocker arms from Cooper-Bessemer.
Since Cooper-Bessemer did not classify the rocker arms as safety related components, the licensee dedicated the newly procured rocker arms to safety related status after receipt onsite.
The replaced components operated satisfactorily during completion of the engine analysis and surveillance test of the 3A EDG.
The 3A EDG was restored to service on January 12, 1989.
Unit 3 returned to Mode 2 (startup) operation
t
on January 22, 1989, following completion of additional short notice outage wor k.
The inspector closely followed the licensee's actions and made the following observations.
o The licensee obtained onsite vendor support immediately upon failure of the EDG.
When turbocharger damage was discovered, a
management level vendor representative arrived onsite to assist in the root cause of failure determination.
o The licensee's engineering staff, both onsite and offsite, utilized vendor representative assistance while establishing the failure mechanisms for both the rocker arm and the turbocharger.
The resulting evaluation appeared to be thorough and complete.
o The Unit 3 work control organization effectively coordinated the licensee's engineering, operations, and maintenance resources to an expeditious return of EDG operability.
o The inspector reviewed the maintenance work package during the repair.
It appeared to be well documented and had appropriate Quality Control (QC) checks and retest requirements.
The licensee and the vendor both issued
CFR 21 reports.
The licensee's report, LER 530/89-004 committed to provide a
supplemental report which will include a Cooper-Bessemer evaluation of the root cause of the manufacturing defect and long term Cooper-Bessemer and licensee corrective actions.
These items wi 11 be followed under the routine LER review program.
No violations of NRC requirements or deviations were identified.
14.
Entr Into Mode 5 (Cold Shutdown) - Unit 3 71707 Unit 3 entered Mode 5 shutdown cooling operation on January 7, 1989, for the first time since achieving criticality.
The inspector conducted a detailed review of shift operations for several hours as the unit entered Mode 5.
During this time the inspector noted the following.
o Key plant management personnel, including the plant manager, were present in the plant.
o Control room operators generally conducted shift operations in a methodical and formal manner, with concern for the new and unfamiliar plant conditions.
o Control room procedures and drawings were current and in use, with steps signed off in a timely manner.
In addition, the inspector noted the following areas of concer The inspector found a 90-95 mr/hr"spot within a posted Radiation Area ("B" Train 40 foot SI pipe chase)
at 1615, approximately eleven hours after initiation of shutdown cooling on "B" Train; The inspector informed the Radiation Protection Manager, and the area was immediately re-surveyed and conservatively posted as a High Radiation Area, even though this posting is normally reserved for dose rates greater than 100 mr/hr.
The inspector surveyed other areas affected by shutdown cooling operation.
No further posting concerns were identified.
The technical basis, as documented on CCI 89-014, was to preclude the possibility of an explosive mixture of hydrogen and oxygen forming in a localized area, which could become hazardous if the RCS were opened for maintenance.
The inspector considered this to be an adequate basis for deviation from the hydrogen specification since there was no intent to open the RCS for maintenance.
However, the inspector noted that the level of plant management authorized to allow this deviation had not, in fact, been given the opportunity to approve it prior to implementation.
Procedu're 74AC-9CY04 Rev.
0 "System Chemistry Specifications" appeared confusing with respect to Reactor Coolant System (RCS)
hydrogen limits required for Mode 5 entry and prior to securing Reactor Coolant Pumps (RCPs).
Hydrogen concentration was given an upper limit of 5.0 cc/kg prior to Mode 5 entry primarily to ensure timely and efficient degassing of RCS coolant.
This limit was waived, by procedure, if Mode 5 entry was forced by a Limiting Condition for Operation (LCO) action requirement.
Separately, however, hydrogen concentration was required to be less than 5.0 cc/kg prior to securing RCPs based on preventing hydrogen accumulation in the RCS (which could pose a personnel hazard from an explosive mixture of hydrogen in air if the RCS were subsequently opened).
No waiver of this requirement existed for a shutdown forced by an LCO action commitment.
At the time Mode 5 was entered (and RCPs secured shortly afterward),
the most recent hydrogen sample was 15.0 cc/Kg.
The Mode 5 entry was forced by an LCO action commitment, and the operators waived the hydrogen limit prior to securing RCPs even though this waiver was not explicitly provided for in the procedure.
The inspector questioned the appr'opriateness of this action without prior management approval since procedure 74AC-9CY04 required that any deviation from these limits received prior approval from the Unit Mater Chemistry Manager (WCM) and concurrence of the Plant Manager (PM) and documentation of the technical basis on a Chemistry Control Instructor (CCI).
At the time the inspector questioned this action, a CCI had not been written, however one was subsequently written and approved by the Unit 3 WCM and PM.
The inspector considered that although plant management had not give prior approval to securing the RCPs with greater than 5.0 cc/Kg, the security of RCP's is closely allied with Mode
!
entry, and the technical basis for the limit was related more to future plant operations (i.e.
opening the RCS) than to any immediate concern requiring prior management approval.
The inspector considered that although management had not given prior approval to securing the RCPs with greater. than 5.0 cc/Kg, management was informed shortly afterwards such that higher than specified hydrogen concentration would be considered, if necessary, duri ng any subsequent evolution involving opening the RCS.
Furthermore, gas stripper operation continued and hydrogen concentration was sampled approximately five hours later, and was 4.0 cc/kg.
The licensee initiated a change request for the procedure to explicitly provide a
waiver option for the hydrogen limit prior to securing the RCPs just as was afforded the limit prior to Mode 5 entry.
The inspector noted that any such waiver should require management notification if these limits were exceeded so that subsequent plant operations could be conducted accordingly.
The licensee acknowledged this concern and committed to addressing it in a procedure change.
(Inspector fol 1 owup item 530/88-41-01).
15.
Char in Pum s Block Cracks - Units 1 and
71707 During recent months several significant charging pump operability problems were experienced.
The Unit 1 "B" charging pump developed a
crack in the middle cylinder of the water block.
The failure was similar to other previously cracked blocks experienced at this plant.
The block was of the original design with exception that the sharp cylinder edges had been rounded and peened to reduce the effects of stress.
The new block used to replace the cracked unit was made with improved strength 17-4 PH stainless steel and was the last spare onsite.
Prior to this replacement an alignment problem between the power drive assembly and the water block on another Unit 1 charging pump caused excessive cylinder wear requiring replacement of both the water block and power drive assembly.
The water block was also replaced with one with improved material strength.
At this time all three charging pump water blocks at Unit 1 are made of the higher strength 17-4 PH stainless steel.
A recent charging pump problem at Unit 3 involving abnormal running noise was determined to be caused by valves which did not appear to seat completely.
This condition was corrected.
If multiple cracked charging pump problems were to be experienced before replacement blocks are received, plant operability may be affected.
Technical Specifications require at least 2 charging pumps in Modes 1-4 for reactivity control.
The licensee is considering the installation of a centrifugal charging pump with a 90 gpm capacity as a long term corrective action.
No violations of NRC requirements or deviations were identified 17.
Dro ed Part Len th Control Element Assemblies PLCEA's) - Unit 3 93702
Il I
V t,
II l
l
On January 26, 1989, Unit 3 dropped Part Length Control Element Assembly (PLCEA) No.
32 while moving PLCEA's for Axial Shape Index (ASI) control.
The PLCEA dropped when its individual power supply breaker opened on over current.
The breaker had been manually cycled the previous day while troubleshooting stuck PLCEA No.
within the same subgroup.
The licensee was able to reclose the breaker and return the PLCEA to its group position.
Reactor power was reduced to less than 80K in accordance with Technical Specifications and was restored to 100K following recovery of the dropped PLCEA.
Licensee troubleshooting primarily involved measuring time dependent current response during PLCEA motion.
On January 27, 1989, another PLCEA, No. 50, dropped when its individual power supply breaker opened.
The PLCEA was recovered in the same manner as the day before and troubleshooting identified gripper coil voltages higher than the values specified in the technical manual.
Higher than normal voltages were identified on four different subgroups which included both PLCEA's which dropped.
The licensee concluded that higher than normal voltages were the most likely cause of an overcurrent condition during PLCEA motion which tripped the PLCEA breakers.
The voltages on four subgroups were adjusted back to their normal bands on January 28, 1989.
They further concluded that the accuracy of the time-current.
response data was inadequate for the licensee to have detected a cumulative overcurrent effect resulting from overlapping operation of the Control Element Drive Mechanism (CEDM) during troubleshooting of PLCEA No.
32.
The inspector made the following observations:
o Control room logs and records for both dropped PLCEA events appeared to be complete and showed compliance with applicable procedures and Technical Specifications.
o The inspector concluded that the troubleshooting of PLCEA No.
32 was incomplete, since no root cause was firmly established and no action was taken which would have precluded dropping PLCEA No.
50.
o No preventive maintenance (PM) checks were in place to periodically test critical CEDM parameters such as voltages and currents to ensure changes due to component aging, temperature, and other factors do not adversely affect CEDM operation.
The licensee indicated that a
PM for checking CEDM parameters would be considered (Inspector followup item 530/88-41-01).
18.
Review of Licensee Event Re orts - Units
2 and
90712 92700
The following LERs were reviewed by the inspector.
Based on the information provided in the report, it was concluded that reporting requirements had been met, root causes had been identified, and corrective actions were appropriate.
The below listed LERs are considered close ll
1 i
l
I t
i I
I l
Unit 1 LER NUMBER DESCRIPTION 88-17-LI Essential chilled Water System Inoperable for nine days in Mode l.
See Section 2.c. of this report for details.
LER NUMBER DESCRIPTION 88-21-LO Reactor Trip Due to High Pressurizer Pressure.
The report discussed a reactor trip which occurred on August 21, 1988, due to high pressurizer pressure.
The high pressurizer pressure condition occurred when steam bypass control valves (SBCV)
1001 and 1004 closed faster than expected and valve 1003 malfunctioned.
These valves actuated when the turbine tripped due to faulty internal contacts in the microswitch that is part of the stator cooling flow pressure switch.
Steam bypass control valve 1003 was disassembled for inspection and rebuilt.
The valve was then exercised several times to accommodate the new packing.
A functional test was then performed to check the quick open and modulation times.
Steam bypass control valve 1003 operated with all quick open and modulation times within the acceptance criteria.
A temporary modification was made to the pneumatic actuators of SBCV 1001 and SBCV 1004 that will allow the modulation time to be set.
A permanent modification to the valves consisting of new pilot valves, which will assist in a faster bleed off of steam that will reduce the speed of valve closure will be tested.
If successful the modifications will be made to all the steam bypass valves at all three units.
The inspector considered the actions taken by the licensee to be adequate.
LER NUMBER DESCRIPTION 88-24-LO Reactor Trip Due To Low Steam Generator Level.
On August 27, 1988, Palo Verde Unit 1 was in Mode 1 at approximately 12K power when a reactor trip occurred due to a low level in steam generator No.
1.
The cause of the event was mainly due to a lack of coordination in controlling reactor temperature.
The control room operator attempted to mitigate the overcooling by increasing the dilution rate.
The shift supervisor directed control element assemblies to be withdrawn and the generator load to be decreased.
While the actions taken were correct, the magnitude of these actions was more than was required to control temperature and overcooli'ng occurred.
As part. of the corrective action the licensee added to Procedure 40AC-9ZZ02 Rev.
0 "Conduct of Shift Operations" an instruction that a briefing or "tailboard" meeting should be held prior to the commencement of all major plant evolutions.
Additionally an
evaluation was performed to assess effectiveness of crew teamwork in mitigating the transient.
Several other operating procedures pertinent to this event were revised to improve operator control of the feedwater system.
The actions taken by the licensee should assist the operators in improving cooldown control and minimize reactor trips resulting from low steam generator level.
Based on this discussion Open Item 88-29-01
"Human Performance Evaluation," is also closed.
19.
Review of Periodic and S ecial Re orts - Units 1 2 and
90713 Periodic and special reports submitted by the licensee pursuant to Technical Specifications 6. 9. 1 and 6. 9. 2 were reviewed by the inspector.
This review included the following considerations:
the report contained the information required to be reported by NRC requirements; test results and/or supporting information were consistent with design predictions and performance specifications; and the validity of the reported information.
Within the scope of the above, the following reports were reviewed by the inspector.
Unit'
o Monthly Operating Report for November, 1988.
o Monthly Operating Report for December, 1988.
Unit 2 o
Monthly Operating Report for November, 1988.
o Monthly Operating Report for December, 1988.
Unit 3 o
Monthly Operating Report for November, 1988.
o Monthly Operating Report for December, 1988.
No violations of NRC requirements or deviations were identified.
20.
Exit Meetin The inspector met with licensee management representatives periodically during the inspection and held an exit meeting on February 1,
1989.
During the exit meeting, the inspector emphasized the importance of having appropriate management approval when deviating from procedural requirements.
The licensee was also advised that the NRC would be interested in the engineering action to correct the charging pump problems.. Finally the licensee was informed that the NRC continued to expect appropriate management involvement to reduce problems due to personnel errors.
Licensee representatives acknowledged the inspector's concern '
l I
l
f