IR 05000528/1994009
| ML17310B258 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 05/03/1994 |
| From: | Wong H NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17310B257 | List: |
| References | |
| 50-528-94-09, 50-528-94-9, 50-529-94-09, 50-529-94-9, 50-530-94-09, 50-530-94-9, NUDOCS 9405060089 | |
| Download: ML17310B258 (58) | |
Text
0 APPENDIX U. S.
NUCLEAR REGULATORY COMMISSION
REGION IV
Inspection Report:
50-528/94-09 50-529/94-09 50-530/94-09 Licenses:
NPF-41 NPF-51 NPF-74 Licensee:
Arizona Public Service Company P. 0.
Box 53999, Station 9082 Phoenix, Arizona 85072-3999 Facility Name:
Palo Verde Nuclear Generating Station Units 1, 2, and
Inspection At:
Maricopa County, Arizona Inspection Conducted:
February 15 through March 28, 1994 Inspectors:
K. Johnston, Senior Resident Inspector H. Freeman, Resident Inspector J.
Kramer, Resident Inspector A. MacDougall, Resident Inspector D. Acker, Reactor Inspector Accompanying Personnel:
J. Ganiere, NRR Intern v
Approved By:
(~
ong, se
,
roJect rane Ins ection Summar ate Areas Ins ected Units
2 and
Routine, announced, resident inspection of:
~
'lant events (93702)
Plant activities and operational safety (71707)
Maintenance activities (62703)
Surveillance activities (61726)
Emergency diesel generator maintenance (62703)
Instrumentation and Control re-engineering pilot program'62703)
High Intensity Training (41500)
Corrective actions for violations (92702)
Inspection followup items (92701)
Licensee Event Reports (90712)
9405060089 940503 PDR ADQCK 05000528
l l
i l
l I
f
Results Units
2 and
Strengths:
Both maintenance and quality control personnel demonstrated good work practices during the replacement of the high pressure fuel injection line on Emergency Diesel Generator 2B (Section 4.2).
The contractor performing the Unit 2 steam generator (SG) tube removal demonstrated professionalism, good communications, and management involvement throughout the effort.
Additionally, radiation protection personnel monitoring the work demonstrated good judgement regarding personnel safety (Section 4.3).
The Unit 2 SG crevice flush and boric acid soak, an evolution performed for the first time on-site, was'ell planned and conducted safely in accordance with procedures (Section 4.4).
~
.
Licensee engineering personnel demonstrated initiative and conducted a
thorough evaluation of main steam safety valve testing issues (Section 5. 1).
The high intensity training for licensed operators was well implemented.
The program featured challenging scenarios, in-depth critiques, and management involvement.
Operators demonstrated good command and control and excellent communications (Section 8).
Weaknesses:
O.
Postmaintenance testing following the gear replacement in a Unit 2 safety-related motor operator did not identify that the gears had been installed incorrectly, causing the valve to operate significantly faster and with substantially less thrust (Section 2. 1).
Engineering and plant operations accepted water hammer events that resulted from opening the steam generator blowdown isolation valves.
The impact of the water hammer events on system components had not been thoroughly evaluated and a design change to correct the problem had not been pursued (Section 2.2).
A cooling line for a Unit 2 emergency diesel generator (EDG), which had been previously abandoned in place, was not adequately supported and was rubbing against an instrument line (Section 3. 1).
Several racked out load center breakers were found to not be in the
. fully racked out position as required by plant procedures (Section 3.2).
A plant modification involving the instrumentation for the essential chiller system was found to not be fully implemented.
Final Safety
t i,
'
r
fr
l i
-3-
I Analysis Report changes had not been made and the annunciator response procedures did not reflect the modification (Section 3.3).
Unit 2 operators misaligned a valve, which resulted in the inadvertent draining of reactor coolant system inventory to the refueling water tank while,establishing a bubble in the pressurizer (Section 3.4).
Contract personnel and licensee engineers failed to follow a step in the procedure governing main steam safety valve testing (Section 5. 1).
~
An evaluation of a damaged valve motor operator torque switch did not address how the as-found torque switch setting had shifted (Section 10.3).
Summary of Inspection Findings:
~
Inspection Followup Items 529/9409-01 (Section 2. 1)
and 528/9409-02 (Section 2.2) were opened.
Two noncited violations were identified (Sections 3.2 and 5. 1).
Unresolved Item 529/9409-03 was opened (Section 3.4).
Violation 529/9311-02 was closed (Section 9. 1).
~
Inspection Followup Items 528/9340-03, 528/9343-02, 529/93-43-05,and 530/9348-03 were closed (Section 10).
/
~
Unresolved Item 528/9401-01 was closed (Section 10).
~
Licensee Event Reports 528/93-02, Revision 0, and 528/93-05, Revision 0, were closed (Section 11).
Licensee Event Reports 528/93-09, Revision 1;
.
529/93-02, Revision 2; and 529/93-05, Revisions 0 and 1, were closed (Section 12).
Attachment:
Persons Contacted and Exit Heeting
l
DETAILS
PLANT STATUS 1.1 Unit
Unit 1 began the inspection period in Mode 1 at 85 percent power.,
On February 28, 1994, the licensee raised the administrative limit on reactor power to 86 percent.
Unit 1 operated at essentially 86 percent power until March 4, 1994, when the unit reduced power to 50 percent to induce a change in SG chemistry (hideout return).
Power was raised-back to 86 percent on March 6.
The licensee continued to monitor a small primary-to-secondary leak in SG 12 that was first identified on February 8.
The leak rate remained less than 1 gallon per day; 1.2 Unit 2 Unit 2 began the inspection period in Mode 5 during a midcycle SG tube eddy current inspection and chemical cleaning outage.
The licensee found a higher than expected number of midspan axial cracks in the SG tubes and plugged
tubes in SG
and 370 tubes in SG 22.
In addition,
SG tubes were cut and removed from SG 22 for further metallurgical evaluation.
On March 21, 1994, the reactor was made critical and tied to the grid on the following day.
The plant held power at 29.5 percent due to secondary chemistry.
On Harch 25, the plant was shut down to repair an oil leak on a
reactor coolant pump.
On March 26, the repairs to the reactor coolant pump were complete and the reactor was made critical.
On March 27, the plant was tied to the grid.
At the end of the inspection period, the licensee was raising power to 85 percent.
1.3 Unit 3 Unit 3 began the inspection period in Mode 1 at 85 percent power.
The unit raised power to 86 percent on February 28 and remained essentially at 86 percent power until the unit commenced a shutdown for the fourth refueling outage.
The unit entered Node 3 operations on March 19 and ended the inspection period in Mode 6.
Pri,or to shutdown, the licensee tested the lift setpoint of 13 SG safety valves (Section 5.2) using a revised "Trevitest" method.
The testing was suspended when the unit shut down for the refueling outage on March 18.
ONSITE RESPONSE TO EVENTS (93702)
2. 1 Hi h Pressure Safet In 'ection Valve Failure - Unit 2 On March ll, 1994, two of the Train A high pressure safety injection (HPSI)
loop injection valves (SIA-617 and SIA-627) would not fully close during motor-operated val've (HOV) differential pressure (DP) testing.
The DP testing
i I
was performed after the installation of new strain gages on the valve stems.
The new strain gages were installed to eliminate potential inaccuracies with the older style strain gages and to provide a more accurate thrust indication.
The inspector reviewed the licensee's troubleshooting efforts, observed the retest of the valves, and discussed the corrective actions and safety significance of the event with the licensee.
The inspector concluded that the licensee appropriately corrected the problem and satisfactorily retested the valves.
However.,
the inspector also concluded that Valve SIA-627'ay'not have been able to shut during the last operating period under design DP conditions due to a maintenance error.
Additionally, the maintenance error had not been identified during the postmaintenance and ASME Code Section XI valve testing.
2. 1. 1 MOV Gears Reversed The licensee determined that the motor pinion gear and worm gear were reversed (the worm gear was in. the place of the pinion gear and vice versa)
in Valve Operator SIA-627.
This caused the stroke time of the valve to be approximately one-half the normal time and prevented the operator from developing sufficient thrust to shut the valve against design DP.
The gears
~
~
~
~
~
were subsequently reinstalled in their proper positions and the valve was successfully tested.
The licensee conducted a review of all maintenance work orders on Valve Operator SIA-627 and determined that the motor pinion gear and worm gear were replaced to correct an abnormal noise in April 1993.
Improper installation was not identified during the postmaintenance testing because the test acceptance criteria of actuator thrust was met.
However, in May 1993, the MOV diagnostic data clearly showed that the valve stroke time was significantly shorter than the previous results (3.74 vs 7.32 seconds).
This difference was not evaluated because the valve stroke time was not included as a test acceptance criteria.
The inspector was concerned that the postmaintenance testing had not identified the error and will evaluate the adequacy of the postmaintenance testing in a future inspection (Followup Item 529/9409-01).
The inspector reviewed the ASME code requirements and the licensee's inservice testing (IST) plan for stroke timing of power-operated valves and determined that there was not a requirement to evaluate stroke times that are significantly lower than the reference value.
The licensee was in the process of reviewing the IST plan and developing appropriate screening criteria for evaluating test results that are significantly lower than the reference values.
As an interim action, the IST technicians were instructed to identify test results that were significantly lower for further evaluation.
The inspector concluded that these actions were appropriate.
The licensee reviewed the maintenance history and previously measured stroke times for valves with the type oF actuators used on SIA-627 (Limitorque Model SMC-04).
The licensee concluded that no other operators were susceptible to this error for the following reasons:
I I
~
Only the Model SMC-04 actuator has motor pinion gears and worm gears that were interchangeable.
~
No other Model SMC-04 actuators have had the worm and motor pinion gears replaced.
~
No other valves had stroke times significantly shorter than anticipated.
The licensee init'iated condition report/disposition request (CRDR) 2-4-0116 to evaluate the safety significance of the problem with Valve SIA-627 and to determine whether the event was reportable.
The licensee concluded that the failure of Valve SIA-627 to close was not safety significant.
The basis for the design DP test in the closed direction was the emergency operating procedure steps which direct operators to close the HPSI discharge valves once the proper reactor coolant system (RCS) level is restored.
The licensee determined that turning the HPSI pumps off and taking manual action to close the valve would have satisfied the function to shut off HPSI flow.
The primary safety function of the HPSI.discharge valves is to open during an accident condition.
The licensee had successfully tested the valves against design DP in the open direction, the direction in which system pressure provides an assist to the valve operator.
The. inspector reviewed the design basis for the HPSI discharge valves and concurred with the licensee's conclusions.
2. 1.2 Strain Gage Inaccuracies The licensee concluded that Valve SIA'-617 failed to shut because the actuator
. torque switches were set up using a measured thrust value that was less than the actual thrust needed to shut the valve.
The licensee determined that the inaccuracies in the thrust values were caused by using a strain gage that was attached to the threads of the valve stem.
Teledyne, the manufacturer of the strain gages, concluded in Technical Report TR-A722-3G that the thrust measurement using the strain gages attached to the threads of the stem resulted in a measured thrust less than the actual thrust.
Teledyne recommended installing the strain gages directly to the smooth portion of the valve stem to minimize these inaccuracies.
The licensee evaluated the potential impact of the strain gage inaccuracies in CRDR 9-4-0021.
In the CRDR, Valves SIA-617 and SIA-627 were identified as needing an additional review to determine the acceptability of the initial DP test data.
The additional DP test data using the new strain gages on the Unit 2 HPSI discharge valves was intended to be used in this evaluation.
The evaluation was scheduled for completion in May 1994.
In the interim, the licensee calculated a higher thrust value using the new strain gages.
The actuator torque switches for Valves SIA-617 and SIA-627 were set up using these higher thrust values and, subsequently, successfully DP tested in the closed direction.
The inspector concluded that the licensee's actions to resolve the strain gage inaccuracies were appropriat I I
I l
2.2 Water Hammer in SG Blowdown Line Unit
0 2.2. 1 Description of the Event On February 14, 1994, during a routine tour of Unit 1, the inspector witnessed a water hammer in a 6-inch, SG 12 blowdown line located in the turbine building.
Containment isolation Valves UV-500R and UV-500S were closed and the downstream blowdown line was drained and depressurized for maintenance purposes.
The water hammer occurred when a reactor operator initiated blowdown flow to the blowdown flash tank by opening the containment isolation valves from the control room with the downstream blowdown piping depressurized.
One pipe hanger support and some thermal insulation on the blowdown piping were damaged as a result of the event.
The water hammer also caused a section of the blowdown piping to hit an instrument air line.
The inspector was informed that these containment isolation gate valves open by air pressure and do not have manual hand wheel operators that-would allow the valves to be opened slowly.
In the procedure for initiating normal blowdown from SG 12 to the blowdown flash tank (410P-ISG03),
Step 12.3.9 cautions that the blowdown system may experience severe water hammer when initiating blowdown flow if the containment isolation valves are closed and the downstream lines are drained.
As a result, the reactor operators anticipated that pneumatically opening the containment isolation valves could cause a water hammer.
Two auxiliary operators were positioned in the turbine building to watch the water hammer during the evolution.
2.2.2 Previous Mater Hammer Events Engineering informed the inspector that there has been a long history of water hammer events in the SG blowdown.lines, which have resulted in repeated damage to hangers and supports.
At the exit meeting, the inspector noted that corrective actions have been focused towards strengthening the hangers and supports, rather than in preventing the water hammer directly.
The inspector also noted that the effect of the previous water hammers on the system was not fully evaluated by the licensee.
2.2.3 Licensee Actions The licensee initiated a
CRDR (1-4-0063) to review the problem of reopening the blowdown isolation valves with the blowdown system depressurized.
The inspector will review the results of the CRDR and the licensee's evaluation of the previous water hammers'ffects on the system in a future inspection report (Followup Item 528/9409-02).
OPERATIONAL SAFETY VERIFICATION (71707)
The inspectors performed this inspection to ensure that the licensee operated the facility safely and in conformance with license and regulatory
.
requirements and that the licensee's management control systems effectively discharged the licensee's responsibilities for safe operatio I
'
J The methods used to perform this inspection included direct observation of activities and equipment, observation of control room operations, tours of.the facility, interviews and discussions with licensee personnel, independent verification of safety system status and Technical Specifications (TS)
limiting conditions for operation, verification of corrective actions, and review of facility records.
3. 1 Abandoned Diesel Generator Line Unit 2 On February 23, 1994, the inspector noted an abandoned 3/8-inch governor cooling water line that appeared to be rubbing against an active 1/4-inch lube oil line near the governor on EDG 2A.
The licensee removed the abandoned section of pipe at th'e nearest union on the engine frame during an earlier EDG outage.
The licensee performed a walkdown of all six EDG's and discovered the cooling water tubing in various states of abandonment.
The licensee also found that the technical manual drawing that had not been changed to reflect the current configuration.
The licensee initiated a
CRDR to evaluate the inspector's finding.
In the CRDR, the licensee identified long term actions to remove the abandoned cooling water tubing on all six EDG's and to change the technical manual drawing to reflect this configuration.
The licensee removed a section of the abandoned cooling water line and determined that the lube oil line had not been damaged by the rubbing action of the cooling water line.
Although the inspector observed wear damage on the abandoned cooling water line, the inspector agreed with the licensee's conclusion that the active lube oil line had not been damaged.
The inspector noted the timely and thorough response of the licensee when informed of the potential p'roblem.
3.2 Load Center Breaker Units
2 and
The inspector noted several instances were the licensee failed to follow procedures when racking out load center breakers.
Fo'r example, on February 24, 1994, the inspector observed a charging pump breaker in Unit 3 not fully racked out (closing springs not discharged)
with a clearance tag attached to the breaker.
The inspector notified the shift supervisor of the problem.
The shift supervisor took prompt action and had the breaker racked out approximately one more turn until the closing springs discharged.
In addition, the licensee wrote a
CRDR to address the problem.
The inspector reviewed the licensee's procedures covering the proper position of a racked out breaker.
Procedure 430P-3PGOl, Appendix J, directed operators to ensure that the closing springs are discharged.
In addition, this procedure stated that, if the racking shutter cannot be closed, the breaker is not in the correct position and to move the breaker either in or out until the shutter closes.
On February 28, the inspector checked that vital load center breakers in all three units were properly racked out.
The following number of racked out
l I
l
breakers were incorrectly racked out:
Unit
2 of ll; Unit 2 8 of 16; Unit 3 - 8 of 12.
The respective unit shift supervisors were informed that racking shutters were open on these breakers.
The shift supervisors had the breakers repositioned to the correct position allowing the racking shutter to close.
Operations management made an entry into the night order book indicating management expectations for operators to correctly rack out breakers.
In addition, a
CRDR was written to address this problem.
The licensee was continuing to evaluate electrical maintenance requirements for removing racked out breakers, the maintenance and operations interface when replacing breakers after maintenance, and auxiliary operator training in racking out breakers.
The inspector concluded that, although the licensee failed to fully rack out the breakers, the safety significance was low.
In all cases the breakers were near the disconnect position.
The breakers could not be inadvertently reconnected or allowed to drop from the cubicle without operator action.
The failure of the licensee to follow procedures is a violation of TS 6.8. 1.
This violation is not being cited because the criteria specified in Section VII.B of the Enforcement Policy were satisfied.
3.3 Essential Chilled Water S stem Ex ansion Tank Pressure and Level Alarms Units
2.
and
During a system walkdown of the essential chilled water system in all three units, the inspector noted that some nonsafety-related instruments were isolated.
These were associated with the essential chilled water expansion tank pressure and level alarms to the control room and a local level gauge glass.
The pressure and level alarms and local level gauge glass are provided for system leak detection and are described in the Updated Final Safety Analysis Report (UFSAR), Section 9.2.9.2.5,
" Instrumentation Applications."
The inspector'found that, approximately 4 years ago, a design change had been performed on the expansion tank instrumentation under an engineering evaluation request and that changes to the UFSAR description and to the annunciator response procedure had not been fully completed.
The nonsafety-related instrumentation loop is separated from the safety-related piping by excess flow check valves.
Manual isolation valves are located upstream of the excess flow check valves.
By design, the excess flow check valves would close following a rupture of the downstream nonsafety-related piping.
In September 1986, the licensee initiated Engineering Evaluation Request (EER) 86-XH-046 to evaluate the use of excess flow check valves as isolation devices between safety-related and nonsafety-related instrument piping.
The evaluation concluded, in part, that, in the event of a rupture in the essential chilled water nonsafety-related instrumentation loop, the excess flow'check valves could fail open, causing an insufficient water level in the system.
The final disposition of the EER required closing the normally open isolation valves in order to isolate the excess flow check valves, In addition, the EER compensatory actions required opening these isolation valves every 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> to obtain a reading of the
'
l(
I l
-10-expansion tank level from the nonsafety-related local level gauge. glass.
The nonsafety-related instrumentation loop and its associated alarms and indication have been isolated for approximately 4 years.
n The inspector noted that, while the licensee had revised the piping and instrumentation diagrams contained in the UFSAR to reflect the new valve'ositions, they had not updated the UFSAR description of the instrumentation.
II In addition, the inspector noted that, while the licensee had revised the essential chilled water system operating procedures, they had not revised the alarm response procedures to reflect that the expansion tank pressure and level alarms were normally isolated.
Several reactor operators, when questioned by the inspector, were unaware that the expansion tank pressure and level alarms were isolated.
In response to the inspector's findings, the licensee initiated CRDR 9-4-0157 to evaluate the adequacy of EER 86-XH-.046.
The licensee determined that engineers had failed to identify the design change as a temporary plant change.
Additionally, the licensee found that a permanent change to use an alternative isolation device had been put on hold.
They concluded that improved design change and plant modification practices would prevent similar situations from occurring at the present time.
At the end of the inspection period, the licensee was evaluating permanent corrective actions.
They were considering implementing a permanent modification versus permanent implementation of the actions originally considered as compensatory.
In the interim, the licensee revised the alarm response procedures to note that the pressure and level alarms were normally isolated.
At the exit meeting, the inspector expressed the concern that neither plant operations nor system engineering had identified that this plant modification was incomplete.
The Director of Site Technical Support discussed this matter with system engineers and emphasized their responsibility in ensuring the resolution of longstanding issues.
In addition, he discussed management's expectations that system engineers should be identifying these types of problems.
3.4 Inadvertent Reactor Coolant S stem Drain to the Refuelin Water Tank-Unit 2 On Harch 12, 1994, operati,ons personnel inadvertently drained approximately 20,000 gallons of the RCS inventory and RCS makeup water to the refueling water tank (RWT) through an open low pressure safety injection (LPSI)
recirculation valve while the plant was in Hode 5 and they were attempting to draw a pressurizer bubble and raise RCS pressure.
On Harch ll, 1994, Unit 2 operators performed engineering safety features actuation system testing that required the LPSI recirculation valve to shut.
Operators realigned the system after the testing was completed.
Although the
fi g
j l
f
-11-realignment procedure instructed the operator to leave the valve in the
"desired position," the operators who realigned the LPSI system could not later explain why they had left the valve in the open position.
There was not a problem at that time in leaving the valve open since a check valve prevented flow from the RWT into the RCS, Train A of shutdown cooling was not in service, and RCS pressure was too low to overcome the RWT head.
On March 12, the licensee began to draw a bubble in the pressurizer.
When the pressurizer heaters were energized to raise pressure, the expansion of the water in the pressurizer overcame the RWT head and the RWT level started to increase.
Pressurizer level was decreasing without the expected increase in pressure.
The licensed operators initially assumed the reason for the lack of pressure increase, pressurizer level decrease, and lack of letdown was due to compressing the air in the SG tubes.
They were not aware that the RWT level was increasing.
After several hours of attempting to draw a bubble, the operators suspected that a problem existed.
However, they continued to conduct the evolution while investigating other causes of the problem.
As part of their investigation to determine where the water was going, the operators monitored tank levels and found that the RWT level had increased 3 percent (20,000 gallons).
This level increase corresponded to the decrease in pressurizer level (9,000 gallons)
and the charging pump flow (11,000 gallons) while attempting to draw a bubble.
The licensee determined that the open LPSI recirculation valve was the path way to the RWT and closed the valve.
At the time of the discovery, the operators had three charging pumps running, letdown flow isolated, nearly constant RCS pressure, and decreasing pressurizer level.
Once the LPSI recirculation valve was closed, pressurizer level and pressure began to rise.
The licensee initiated a
CRDR and a human performance evaluation of this event.
On March 17, a conference call was held between the licensee and Region IV Walnut Creek Field Office management.
Region IV management expressed a concern regarding the control of significant plant evolutions involving RCS inventory and the necessity for the licensee to aggressively
~
identify and address the cause of this event.
The Unit 2 Plant Manager agreed and stated that they considered this to be a significant concern in the control of RCS inventory.
He identified that several opportunities had been missed to ensure that the LPSI recirculation valve was correctly positioned.
In addition, he noted that the shift supervisor should have been more cautious in evaluating the inventory reduction.
Licensee management stated that these concerns would be addressed in the review of the event.
Subsequently, the inspector identified that the licensee's Emergency Plan Implementing Procedures stated that an event in Mode 5 involving an RCS leak rate greater than 44 gpm be considered an Unusual Event (Emergency Plan Implementing Procedure 02, Revision 12, Appendix B, Table 2).
The inspector noted that the licensee had not declared an Unusual Event.
At the exit meeting, the inspector asked licensee management what gas their conclusion regarding classification of the March 12 event.
The Unit 2 Plant Manager
-12-stated that this issue would be evaluated.
The inspector will review the licensee's event classification evaluation when it is complete (Unresolved Item 529/9409-03).
NAINTENANCE OBSERVATIONS (62703)
During the inspection period, the inspectors observed and reviewed the selected maintenance and activities listed below to verify compliance with regulatory requirements and licensee procedures, required quality control department invol'vement, proper use of safety tags, proper equipment alignment and use of jumpers, personnel qualifications, appropriate radiation worker practices, calibrated test instruments, and proper postmaintenance testing.
Specifically, the inspectors witnessed portions of the following maintenance activities:
4. 1 EDG Bent Connectin Rod Unit 3 Mhile performing outage maintenance on EDG 3B on March 23, 1994, the licensee discovered damage to Connecting Rod 6L (sixth cylinder, left bank, articulated),
Connecting Rod 6R (master),
and the articulated rod pin.
The licensee discovered that the articulated rod had bent, that the articulating rod pin had seized in its bushing, and that the master connecting rod showed signs of heat str ess.
A brief description of the engine and connecting rod layout is provided below.
The EDG is a V-shaped, 20-cylinder, turbocharged engine built by Cooper-Bessemer arranged in two banks of 10 cylinders.
Because the two banks are not offset, power must be transmitted to the crankshaft through a master rod/articulated rod configuration.
The pistons from the right bank are connected to the crankshaft by the master connecting rods in a standard configuration.
The pistons from the left bank are connected, by articulated rods, to pins in the master rod.
The pins allow the articulated rod to pivot relative to the master rod as the crankshaft rotates.
Cylinder 6L of EDG 3B had experienced intake and exhaust valve rocker arm failures on July 28, 1993 (see NRC Inspection Report 50-528/93-35; 50-529/93-35; 50-530/93-35).
The licensee believed that the damage to the connecting rods and bearings was caused by these previous rocker arm failures.
However, the licensee failed to detect this damage during repairs to the rocker arms.
Additionally, diesel engine analysis conducted on August 25 and October 20, 1993, indicated a low compression ratio in Cylinder 6L.
Even though the licensee speculated at the time that the low compression ratio could be caused by a bent connecting rod, they had concluded that it was caused by exhaust blow-by of the compression rings.
The inspector questioned the licensee whether the EDG could have performed its safety function with the damaged components.
At the completion of the inspection period, the licensee had not completed their evaluation.
The inspector will review the licensee conclusion in a later inspection.
Additionally, the inspector will review the licensee's corrective action to
l
0 J
I
I I
l I
f
-13-the rocker. arm failures.
Finally, the inspector will review the licensee's actions from August 25, 1993, through March 23, 1994, in response to indications of component damage.
4.2 Diesel Generator Hi h Pressure Fuel Line Re lacement Unit 2 On February 17, 1994, the inspector observed the installation of the high pressure fuel injection line on the 2B emergency diesel generator (EDG).
The maintenance workers exhibited strong knowledge of the equipment they were working on and the tasks being performed.
guality Control (gC) personnel exhibited active involvement in the installation process of the fuel line.
The inspector concluded the licensee demonstrated good work practices during a
maintenance evolution.
4.3 SG Tube Removal Harvest
- Unit 2 The licensee removed portions of 21 tubes from SG 22 for analysis.
The sections removed were between the upper most horizontal support to the first vertical support and included the bend region.
Most of the harvested tubes will be evaluated by Combustion Engineering and the remainder by an independent laboratory, The inspector observed the whip cutting and removal of the tubes from the Babcox
& Wilcox Nuclear Technologies (BWNT) control trailers.
The inspector noted the professionalism of the BWNT personnel, good communications, and the presence of BWNT management throughout the evolution.
Inside containment, the inspector observed good radiation work practices.
For example, a radiation protection technician demonstrated excellent judgement when removing a
BWNT worker from an SG tube removal platform due to heat stress.
The NRC staff will review the results of the tube harvest evaluation when it is formally submitted.
4.4 Boric Acid Crevice Flush Unit 2 On March 15, 1994, the inspector observed the licensee perform crevice flushing on Unit 2 SGs.
The inspector watched the evolution from the control room, observed the plant's response, and reviewed the procedure.
The inspector concluded that the work was conducted safely and in accordance with the procedure and that the plant responded as expected.
The boric acid crevice flush was designed to reduce intergranular stress corrosion cracking by forcing caustic impurities out of cracks and crevices and replacing them with boric acid inhibitors.
Because the process had not been performed at Palo Verde, the licensee developed a boric acid crevice flush procedure (420P-22Z13)
based on Electric Power Research Institute guidelines.
These guidelines consisted of:
an ambient soak, the crevice flush, a low power soak, and on-line conditioning.
The inspector reviewed the procedure and pointed out a few deficiencies to the project engineer.
For example, one step directed the SG water level to be drained to z 21 percent
r
t
but did not specify the lower limit.
The inspector concluded that these deficiencies did not have an effect on the overall performance of the evolution.
The engineer committed to resolve these and other procedural deficiencies prior to performing the process during the Unit 3 outage.
The inspector concluded that this was appropriate.
SURVEILLANCE OBSERVATION (61726)
The inspectors reviewed this area to ascertain that the licensee conducts surveillance of safety-significant systems and components in accordance with TS and approved procedures.
5. 1 Hain Steam Safet Valve Testin Unit 3 On Harch 8, 1994, the licensee received a letter from the Furmanite company that described a change in the method of calculating the mean seat area (HSA)
used in determining the,lift setpoint of main steam safety valves (HSSV).
Based on this letter and an independent evaluation conducted by plant engineering, the licensee decided to perform on-line testing of the HSSVs.
The licensee had previously suspended on-line testing of the HSSVs using the
"Trevitest" method in September of 1993 due to potential differences between the Westinghouse live steam test and the Furmanite Trevitest method (see NRC Inspection Report 50-528/93-40 for details).
The inspector reviewed the licensee's evaluation of the Furmanite test data, attended a Plant Review Board where licensee management approved the use of the Trevitest method of HSSV testing, and observed on-line testing of several of the HSSVs in Unit 3.
The inspector concluded that the licensee had conducted a thorough evaluation of the test data collected at the Westinghouse Test Facility and had already concluded that there was an offset between the two test methods.
However, the inspector also noted that licensee engineers did riot ensure that the Furmanite representatives conducted the testing in accordance with the licensee's procedure during the testing in Unit 3.
5. l. 1 HSA Calculation The HSA is used to convert a direct force measurement from the lift mechanism (called a Trevitest) to a lift pressure.
Furmanite revised the calculated HSA value based on a comparison between the actual lift setpoint (determined by the Westinghouse live steam method)
and the setpoint using the Trevitest method based on data from 37 valves.
Palo Verde engineers initiated the tests in September 1993 when they suspected there was an offset between the two test methods.
Licensee engineer's reviewed the Furmanite data and evaluated the impact of the new HSA on previous HSSV test data.
The licensee concluded in a CRDR (9-4-0160) that the number of valve setpoints that were outside the TS limit of +/-
1 percent were reduced from 48 with the old HSA (24.626") to 29 with the new HSA (23.046").
The licensee believed that the 29 Trevitest failures were inherent to the specific valve and not due to the testing difference s'
I J
,~
l H
-15-Based on this information, the licensee was confident that they could conduct on-line testing of the HSSVs using the Trevitest method with the new HSA value and directly compare the results with the a'-left Westinghouse test method setpoints.
To further validate their decision, licensee engineers conducted a
CFR 50.59 evaluation of the Trevitest method and asked the Independent Safety Evaluation Group to conduct a review of the test data supporting this recommendation.
The Independent Safety Evaluation Group and the Plant Review Board agreed that performing on-line testing of the MSSVs in Unit 3 with the new HSA was technically sound based on a significant amount of data (a total of 228 Westinghouse and Trevitest lifts of 15 Palo Verde valves).
The inspector agreed that the decision was based on sound engineering data..
5.1.2 Lift Test Observation On March 17, the inspector observed licensee engineers and Furmanite representatives conduct on-line testing of the HSSVs using the Trevitest test method in Unit 3.
The test was conducted using Procedure 73AC-9ZZ18,
"HSSV Set Pressure Verification."
The inspector noted that the Furmanite representative did not verify the correct position of two valves on the test device prior to moving the operator control valve to the lift position.
The procedure stated that these steps were to be performed.by the Furmanite representative and verified by the licensee test director.
The inspector also noted that the test director was not verifying the completion of these steps as directed by the procedure.
The inspector concluded that the licensee failed to follow Procedure 73AC-9ZZ18.
The inspector described the situation to the mechanical engineering supervisor who stated his expectations that the test should be conducted in accordance with the procedure.
The supervisor discussed his expectations with the test directors and established measures to verify the positions of the test device valves and ensure procedure compliance.
The inspector concluded that the failure to verify the positions of the valves would not have affected the measured lift setpoint of the valves.
The failure of the licensee to follow procedures is a violation of TS 6.8. 1.
This violation is not being cited because the criteria specified in Section VII.B of the Enforcement Policy were satisfied.
5.1.3 Lift Test Results The licensee tested 13 of the 20 HSSVs.
Six of the valve setpoints were within the TS limit of +/-
1 percent.. The setpoints of another six valves, were within a,wider band of +/- 3 percent and were reset to within the TS limit of +/-
1 percent.
The licensee had previously performed an evaluation which determined that the safety valves which operated within this wider band could accomplish their safety function.
The setpoint of one other valve was within the +/- 3 percent, but the'licensee was unable to reset the valve to within the +/-
1 percent limit.
This valve and the other seven valves that
',)
fl k
'
-16-were not tested were scheduled to be sent to the Westinghouse facility to be refurbished and retested.
The inspector noted that the number of valve setpoints within the +/-
1 and 3 percent limits had increased compared to previous tests.
5.2 Other Surveillances Observed n,
42ST-2SIIO High Pressure Safety Injection Pump Operability Test - Unit 2
EMERGENCY DIESEL GENERATOR MAINTENANCE PROGRAM (62703)
The inspector conducted a detailed review of licensee procedures and documents, interviewed the EDG system engineers, and observed a planned outage on the Unit
EDG B to assess the effectiveness of the overall EDG maintena'nce program.
6.1 ~A1 R
The'inspector reviewed the 1993 annual report published by the EDG system engineers.
The inspector noted that EDG reliability was good.
Units 1 and
had only one failure to start in the last 100 attempts and Unit 2 had two failures in the last 100 attempts..
The inspector also noted that the unavailability of the EDGs was low and was within the site goal of less that 1.8 percent.
The inspector reviewed the licensee's resolution of several technical issues related to the EDGs and noted that the licensee had outlined reasonable actions to correct the problems.
For example, a problem with oil carryover from the air compressors was corrected by completing a design change package to replace the compressors.
Appropriate interim actions were established to periodically change critical air system filters to ensure any oil leakage would be trapped and evaluated.
The 'design change package was scheduled for installation during the current Unit 3 refueling outage.
In January 1993, a team of individuals from various site organizations conducted a detailed walkdown of all six EDGs.
The results of the walkdown were documented in the annual report.
After the walkdowns were conducted, the team met with the work planning organization to initiate work documents for any system problems.
The inspector concluded that deficiencies identified during the walkdown were appropriately evaluated and integrated into the maintenance process.
6.2 Preventative Haintenance Basis Review Palo Verde has developed a data base which includes the definition of the bases for performing preventative maintenance (PH) activities.
The inspector reviewed the data base for the following components:
the EDG air start compressor, the lo'w lube oil pressure switch, and the starting air filter.
The inspector concluded that the basis for the PHs considered all available sources of information and that the licen'see had implemented an effective PH
f
-17-basis program.
The inspector noted that, the PH basis for the air start compressor deviated from the, vendor recommended inspections.
This was justified by stating that the operator logs would cover the vendor requirements.
The inspector reviewed the operator logs and verified that the intent of the vendor requirements were covered by completion of the logs.
6.3 EDG Limitin Condition for 0 eration Haintenance The licensee conducted a preplanned maintenance outage of EDG B in Unit 1 on March 8, 1994.
The inspector attended the final planning meeting, reviewed the work scope, and observed several of the jobs in the field.
6.3. 1 Planning The inspector attended the last of four planning meetings prior to the outage.
The inspector concluded that the outage was well planned.
All outstanding work orders were reviewed and only those that would render the EDG inoperable were included in the work scope.
The planners generated a work schedule which showed a total outage duration of 15 hours1.736111e-4 days <br />0.00417 hours <br />2.480159e-5 weeks <br />5.7075e-6 months <br />.
All the work disciplines and operations were, involved in prioritizing the work.
However, the inspector noted that the system engineer and a representative from the PH basis group were not involved in the planning meeting.
The licensee also identified this omission during a critique of the outage and decided to include them during future EDG outages..
The inspector conducted an independent review of all the open work orders for.
system leaks that could be fixed during the outage.
The inspector identified one work order that was not on the final schedule.
The work planner had already identified the work order, but had inadvertently omitted it from the schedule.
The work order was included in the final schedule and the leak was repaired.
6.3.2 Low Lube Oil Pressure Switch Calibration The inspector observed the calibration of the low lube oil pressure switches.
These" are safety-related switches that provide an EDG trip in the emergency mode on a loss of lube oil.
The technicians performing the test noted that the switches were acting erratically and recommended replacing the switches.
The switches were replaced, and the system engineer was notified of the condition.
During the calibration, the inspector noted that the PM task sheet did not have a step directing a
CRDR to be written for any out-of-calibration (OOC)
conditions.
The licensee was previously issued a Notice of Violation in NRC Inspection Report 50-528/92-14 due to a failure to trend OOC conditions with the EDG low lube oil pressure switch.
The inspector noted that the licensee's corrective actions did not include the low lube oil pressure switch in the OOC program.
Based on the inspector's questions, the licensee conducted a review of nine critical EDG instruments and determined that the PH tasks used to conduct the calibrations of seven of the nine instruments did not include
(i t0 I
I))
y-18-steps to ensure OOC conditions were evaluated.
" The inspector concluded that, without the steps to perform evaluations, any "as found" setpoints that were 00C may not have been identified and evaluated.
At the exit meeting, the inspector asked the licensee if they considered that the PH tasks for the seven instruments, without a step to perform an evaluation and determine if any "as left" setpoints were outside the acceptance criteria, were adequate.
Additionally, the inspector asked if previous data was acceptable.
The inspector also asked if the EDG instruments should have been included in the scope of the OOC program.
The inspector's review of this issue will be included in an ongoing inspection of the licensee's OOC program.
6.3.3 Control Air Valve Haintenance P
During the retest of the control air, valve for the fuel rack control cylinder (DGN-UV-0244), the technicians observed air blowing out of a weep hole in the valve, which prevented the control cylinder from being pressurized.
The valve had been removed, disassembled, and rebuilt using new elastomers during a
scheduled preventative maintenance task.
The inspector determined that the failure to pressurize the control cylinder would not have prevented the EDG from starting in the emergency mode.
The licensee rebuilt the valve again with.new material, but did not resolve the problem.
The system engineer identified a "U-cup" that was installed backwards and allowed the pressure to bleed off the control cylinder.
The
"U-cup" was properly installed and the valve was satisfactorily restored to service.
CRDR 1-4-0090 was written to evaluate this problem and to determine corrective actions.
The installation instructions were revised to show the proper 'orientation of the "U-cup" and the event was included in the maintenance department industry events briefing.
The inspector concluded that
.the licensee's troubleshooting efforts and corrective actions were appropriate.
RE-ENGINEERING - INSTRUNENTATION AND CONTROL (IKC) CONTROL OF WORK PILOT TEAN On February 21, 1994, the licensee started the first pilot team for re-engineering the control of work in the IIlC area.
The purpose of the pilot team was to test a streamlined work control process developed through the re-engineering program.
The inspector reviewed the procedure for control of work for the pilot teams, observed a portion of the performance of surveillance test on the plant protection system (PPS),
and interviewed the pilot team leader.
The inspector concluded that the team was keeping up with -the required surveillance tests and emergent work.
The team also was able to more efficiently correct emergent plant problems.
The inspector also noted that
,the work was being effectively scheduled, prioritized, and conducte I if
-19-The inspector noted that the team was responsible for the PPS, the engineered safety features actuation system, and the plant ex-core instrumentation.
The normal workload for these three systems consisted of. about 10 major surveillances a week.
The inspector noted that, although the team was able to perform this work, they were unable to reduce the work backlog.
The inspector also noted that there was a large amount of cross-training that was necessary for the team to become more efficient with backlog reduction.
The team leader stated that, as the team members mastered all the other responsibilities of work performance (writing work packages, documenting completed work, scheduling of work, etc.),
they would have more time to work the backlog.
The inspector concluded that the team would eventually be able to concentrate some effort toward backlog reduction and that management was tracking the backlog to ensure it did not increase.
The inspector noted that the team consisted of six I&C technicians, two planners, a system engineer, and an I&C standards representative.
With this mix of personnel, the team would have ownership of all the processes involved in the performance'f their work.
This enabled the team to more effectively deal with plant problems. 'or example, a problem with PPS testing was resolved within 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />.
With the previous organization, the same'roblem would have taken significantly longer to resolve.
The inspector observed the technicians in the field and noted that there was one fully qualified technician who could work alone (an independent worker)
and the system engineer who was not fully qualified to work alone (a dependent worker).
The inspector noted that the dependent worker was under the direct observation of the independent worker.
The inspector noted that this met the intent of the licensee's procedures for using unqualified workers to perform work.
However, a licensee employee raised a concern involving a situation where the system engineer, a dependent worker, was working independently without any direct supervision.
The licensee initiated a
CRDR (1-4-0089) to evaluate this situation.
In the CRDR evaluation, the licensee determined that existing procedures allowed the use of dependent workers without direct supervision.
However, to allow a dependent worker to conduct maintenance without another independent worker present, the work group supervisor (WGS) must conduct a prejob briefing and monitor the activity based on the WGS's confidence in the worker's experience and qualifications.
In the situation where the system engineer was conducting the maintenance by himself, the WGS had enough confidence in his ability based:
on his experience as a former technician.
The licensee initiated action to clarify the intent of using dependent workers to conduct maintenance based on INPO guidelines.
The inspector concluded that the licensee appropriately resolved this issue.
HIGH INTENSITY TRAINING (41500)
In response to a weakness identified in licensed operator requalification and initial exams, the licensee developed the high intensity training (HIT)
I 'I H
'I l
-20-*
program as a means of increasing operator familiarity with the emergency operating procedures, strengthening the command and control of the shift supervisor and assistant shift supervisor, and improving the overall communications of the crew.
The inspector observed several simulator scenarios and crews during two of the HIT cycles.
The scenarios were challenging and caused the operators to use all available indications to diagnose the problem.
The inspector noted two cases where unit operations management observed the performance of their crews and the lead'hift technical advisor observing the shift technical advisors to ensure uniformity.
The licensee had sufficient training staff in the simulator to evaluate crew actions.
The simulator critiques were thorough, and where questions were raised concerning operator performance, the evaluators would rerun the simulator to verify that the operators and simulator responded as expected.
The inspector concluded that the training staff did a good job in implementing the HIT program.
Additionally, the operators'emonstrated good command and control and showed excellent communications.
FOLLOWUP ON CORRECTIVE ACTIONS FOR VIOLATIONS (92702)
9. 1 Closed Violation 529 9311-02:
Failure to Plu a Defective Steam Generator U-Tube Unit 2
This violation was issued because the licensee plugged the wrong steam generator U-tube and, as a result, failed to plug a defective tube (defect greater than 40 percent)
as required by TS Surveillance Requirement 4.4.4.4b.
Additionally, the licensee's contractor performing the tube plugging had made several other errors during the tube plugging process.
Tube plugging operations were conducted by BWNT using a vendor-developed procedure.
The inspector reviewed the field procedure for tube plugging and determined that the immediate corrective actions taken after the tube misplugging and misinsertion events in Unit 2 were first discovered (NRC Inspection Report 50-528/93-35; 50-529/93-35; 50-530/93-35)
had been temporary and had not been incorporated into the permanent revision.
The inspector reviewed the tube plugging process and procedures and the corrective actions'or the latest plug misinsertion (NRC Inspec'tion Report 50-528/93-55; 50-529/93-55; 50-530/93-55).
During the Unit 3 midcycle outage in December 1993, a quality control (gC) inspector discovered a misinserted plug during the video verification of the plug locations and prior to the tube being rolled in place.
The inspector determined that the licensee actions in response to the violation had focused on ensuring that a misinserted tube plug would be detected prior to being rolled in place.
While gC steps that were incorporated into the procedure were effective, the licensee had not taken steps to address the performance of the BWNT personnel performing the plug insertions or the quality of the procedure they used.
The inspector concluded
c I
l l
l
-21-that the temporary procedure changes used during the Unit 2 refueling outage were not incorporated into the permanent revision and that these ch'anges could have prevented the Unit 3 misinserted plug.
The inspector reviewed the latest version of the tube plugging procedure, 31CP-9RC01, Revision 5.
The inspector concluded that changes made to the procedure would prevent future plug misinsertions, including clarification on the expectations on how to locate a target tube.
However, current licensee imposed requirements on the contractor, such as second person verification, were not incorporated into the procedure.
The licensee assured the inspector that these requirements would remain in effect until the contractor demonstrated that there would be no further misinsertions.
The inspector concluded that the format and organization of the current procedure was confusing.
The licensee agreed that the procedure jumped around and was confusing in some areas.
The procedure was developed by the contractor as a generic procedure and was intended to be used by the contractor at other facilities.
Because contractor employees had used versions of this procedure at other utilities throughout the nation and were familiar with the procedure, the licensee had not rewritten the procedure'o the.their own format prior to approving it for use onsite.
However, the licensee was considering rewriting or having the contractor rewrite the procedure to eliminate the confusion.
The inspector observed tube plugging activities 'during the Unit 2 mid-cycle outage.
The inspector observed that, prior to the insertion of the plug, the plug location was verified to be correct by an individual other than the manipulator operator.
A third individual then verified all inserted plugs and recorded their locations on video tape.
The gC personnel checked the video to verify proper plug location and then independently rechecked
'the video for plug accuracy.
The plugs were then rolled.
Two individuals would independently verify that the correct plug was being rolled.
Normally the inserted plugs have a paper barrier over the end of the plug that would be punctured by the rolling machine.
If the paper barrier was missing, plant gC was required to verify that the rolling machine was in the correct location prior to rolling.
A supervisor checked the computer generated data following the plug roll to ensure that proper roll time, torque, and plug expansion had occurred.
The inspector concluded that the steps taken by the licensee were appropriate.
FOLLOWUP (92701)
10. 1 Closed Ins ection Followu Item 528 9340-03:
E ui ment ualification E
Issues with Tar et Rock Solenoid Valves This item involved the adequacy of the Eg life of the solenoid valves for the steam bypass valves to the turbine-driven auxiliary feedwater pump (AFW)
(SGA-UV-134A and -138A).
This issue also included a review of the licensee's
a
I I
I l
-22-evaluation of the overall reliability of Target Rock solenoid-operated valves (SOVs) located in the main steam support structure (HSSS).
The licensee initiated a
CRDR (9-3-0456) to validate the existing qualified life of the SOYs in the HSSS based on the high ambient temperatures in this location.
The inspector, reviewed the CRDR and noted that only one coil failure occurred on the normally closed (de-energized), Target Rock SOVs.
The licensee also evaluated several reed switch failures and determined that they would not have prevented the valves from performing their safety function to open.
The licensee also took on-line temperature profiles'f Valves SG-134A and -138A to verify the temperature profile assumed during the qualification testing.
Based on this information, the licensee changed the qualified life of the normally de-energized coils from 20 to 40 years.
The licensee'-s guality Assurance department reviewed the disposition of CRDR 9-3-0456 and requested Eg personnel to take additional data in the units.
When this data was taken, the temperature profile was about 100'F higher than the temperature profile used in the Eg life calculation.
The licensee suspected that the valves were leaking which caused the higher temperature readings.
Work orders were written to correct the seat leakage and take another set of temperature readings.
These actions were scheduled during the current refueling outage in Unit 3.
Additionally, temperature readings of Valves SG-134A and -138A will be taken in Unit 2 when it returns to power operation.
The inspector reviewed the temperature data, the Eg calculation, and the failure history of Valves SG-134A and -138A and determined that there was a basis for extending the current Eg life to 40 years.
The inspector also noted that the additional temperature profile information would provide greater confidence that the Eg life of the coils was appropriate.
The inspector also reviewed CRDR 1-1-0055, which was written by the licensee to evaluate solutions to the high failure rate of Target Rock SOVs used in the HSSS.
As of February 1994, there had been a total of 64 coil failures, 10 reed switch failures, and 23 problems with improperly set reed switches.
The majority of the problems were with the SG steam trap isolation valves which are normally open (solenoid energized)
and are required to close during a containment isolation signal.
The licensee engineers concluded that leakage past the steam traps and the steam trap bypass valves increased the heat generated in the isolation valves and accelerated the failure of the solenoids.
A preventative maintenance program was established to identify and correct any seat leakage and operations procedures were changed to minimize the use of the steam trap bypass valves during main steam startup.
In addition, the following long-term corrective actions were planned:
~
The licensee planned to install an SOV from a different vendor in Unit
during the spring 1995 outage and evaluate its performance for a minimum of 18 month c S
I
'
i t
-23-
~
The licensee was evaluating the installation of current control units in the circuit to the solenoid to lower the continuous current level.
This would reduce the effects of self-heating from the coil.
~
The licensee was evaluating the implementation of a new bolted bonnet assembly unit on the Target Rock SOVs, which would acc'ommodate more frequent internal inspections of the valves.
The inspector concluded that the above recommendations should help improve the reliability of the Target Rock SOVs used in the HSSS.
The inspector also noted that the normally energized SOVs fail in the closed position, which is the safety function of the valve.
Therefore, the inspector concluded that the high number of coil failures was not safety significant.
10.2 Closed Ins ection Followu Item 528 9343-02:
Low Pressure Safet In ection Pum Breaker Failure to Close Unit
This item involved the LPSI A pump motor breaker failure to remain closed when operators attempted to start the pump on October 23, 1993.
The licensee's initial evaluation determined that the closing mechanism of the breaker (a General Electric Hagne-Blast breaker)
was not setting properly and that the prop spring mounting bracket was loose.
The inspector reviewed the licensee's detailed root cause of failure analysis documented in CRDR 1-3-0599.
The licensee determined that the most probable cause of the failure was the loose prop spring mounting bracket.
However, during the troubleshooting the licensee also identified that the PH basis program was not properly implemented.
The licensee inspected the position of the prop pin and the tightness of the prop pin mounting bracket on 13 other Hagne-Blast breakers of this size in Unit 1 and did not identify any other problems.
The licensee also conducted a
search of breaker failure data and did not identify any other failures of this type.
An update to the applicable PHs was initiated to require verification of the correct location of the prop pin and the correct tightness of the mounting bracket.
This update was scheduled for completion in June of 1994.
The inspector concluded that the licensee's corrective actions were thorough and should prevent similar problems with loose prop pin mounting brackets.
The inspector also noted that the licensee had previously identified the problems with the PH basis program and had implemented appropriate corrective actions.
10.3 Closed Ins ection Followu Item 529 93-43-05:
AFW HOV Failure-Unit
This item involved the failure of the steam supply valve from SG 22 to the turbine-driven AFW pump (Valve 2SGA-UV-0138) to close using the motor operator.
following a reactor trip on November 1,
1993.
An initial evaluation
< I I
l I
I
-24-determined that the closed torque switch (TQS)
was broken, which prevented the valve from fully closing.
The inspector reviewed the licensee's root cause of failure analysis for the broken TQS documented in CRDR 2-3-0594.
The licensee determined that the failure of the closed TQS was caused by a sudden overload condition that exceeded the yield strength of the TQS (the TQS was manufactured from a low grade aluminum alloy).
The'icensee determined that the types of valves that were susceptible to the high loading condition were the larger gate valves with fast actuating times.
In these particular applications, the valve was suddenly pulled out of the seat, which would cause the motor operator drive train to undergo a tran'sient load.
This load would subsequently be transferred to the TQS, which would cause the TQS to "snapback" from the open side to the closed side.
This problem had previously caused several TQS roll pins to shear, including the roll pin installed in SG 134 in Unit 2 in 1991, and was evaluated in CRDR 2-1-0082.
In this evaluation, the licensee installed strain gages on the TQS, which verified the existence of the high stress caused by the snapback phenomena.
The licensee conducted a review to determine which HOV actuators may have the older aluminum TQSs.
During initial plant startup, the licensee replaced the TQS insulating material in all HOV actuators to correct a potential equipment qualification issue identified by the vendor (Limitorque).
At the time of the changes, the older style aluminum TQSs were not available and the newer brass TQS were installed.
This change was not performed on SG 134 and SG 138 because actuators using a direct current power supply were specifically exempted by Limitorque.
The licensee determined that the brass TQSs had been subsequently installed in Unit 2 SG 134 and Unit
SGs 134 and 138.
Work orders were written to inspect the TQSs on SGs 134 and 138 in Unit 3 during the current refueling outage.'he licensee also initiated a design change to install
"no lost motion" drive sleeves on these type actuators to reduce the inherent TQS transients caused in the fast acting gate valve applications.
The inspector concluded that these actions were appropriate.
The licensee determined that the as found closed TQS setting for SG 138 in Unit 2 had moved from its previous setting of 2 to 3.
The inspector reviewed the licensee's evaluation of this condition documented in Engineering Evaluation (EER) 93-MO-0210 and noted that there was not an evaluation as to why the TQS setting moved and the potential impact of this problem on similar HOVs in the other units.
The inspector concluded that the licensee's evaluation was not thorough.
The licensee agreed with the inspector's observation.
In response to the insp'ector's questions, the HOV engineer determined that the TQS movement was most likely caused by repeated TQS snapback and the older style smooth TQS dial face.
The newer style has a serrated face which would limit the potential motion of the TQS set screws.
The inspector verified that the TQS snapback would cause the closed TQS setting to move from a lower to higher value.
The licensee documented this evaluation in EER 94-H0-026, which superseded EER 93-HO-0210.
The inspector concluded that the revised EER
e l I J
i
~
I l
f I
I
-25-
. appropriately addressed the root cause of the problem and actions to prevent recurrence.
10.4 Closed Ins ection Followu Item 530 9348-03:
Excore Safet Channel Volta e Ram Rate Heasurement This item involved the method of determining a voltage ramp rate measurement in the excore safety channel log calibration surveillance test (36ST-3SE01).
The procedure directed the technicians to use a digital voltage meter and a
hand held stopwatch to adjust the voltage ramp to 7.000 a 0.050 volts dc.
The inspector determined that the procedure was inadequate to achieve this tolerance.
Additionally, the method for determining the ramp rate was unclear.
The licensee documented the deficiency in CRDR 3-3-0466.
The inspector reviewed the CRDR and concurred that the voltage ramp rate adjustment did not affect the safety function of the instrument.
However, the inspector noted that the CRDR did not address the resolution requested by the CRDR initiator - particularly, how should the ramp rate be measured.
The inspector noted that the CRDR had not resolved the problem of the test requiring a tighter tolerance than could be measured using the method provided.
The inspector discussed with the I&C engineer the need to either widen the tolerance band so that it could be achieved using the current method or to change the method.
The engineer agreed and noted that the CRDR required resolution by April 15, 1994.
A foreman in the ILC pilot team informed the inspector that the procedures that adjust the voltage ramp rate (36ST-ISE01, 36ST-2SE01, 36ST-3SE01 and 36ST-9SE12)
were placed on administrative hold and would not be used until the issue was resolved.
Because the voltage ramp rate adjustment does not affect the safety function of the instrument, the inspector concluded that the licensee's actions were appropriate.
10.5 Closed Unresolved Item 528 9401-01:
Shim Control for General Electric GE Han e-Blast Circuit Breakers
This unresolved item identified that licensee procedures did not maintain specific control of shims within GE 4160 volt Hagne-Blast circuit breakers.
These shims were not shown in GE drawings and were found by the licensee to be installed differently in each Hagne-Blast circuit breaker.
The licensee was overhauling these circuit "breakers using a self-developed procedure.
The item was left unresolved for further NRC review of this item for compliance with
CFR Part 50, Appendix A, Criterion III, Design Control.
The inspector noted that shim changes in a GE 4160 volt Hagne-Blast circuit breaker at another site may have caused failure of a circuit breaker to properly operate.
After discussions with cognizant NRC engineering personnel, the inspector determined that the specific requirements of 10 CFR Part 50, Appendix A, Criterion III, Design Control, did not apply to shims within the GE 4160 Hagne Blast circuit breakers since the shims were not shown in GE drawings.
The
r)
g I
I t
I i
'
-26-shims are viewed as maintenance items to be used to ensure proper circuit breaker operation.
Subsequent to the original inspection, the licensee determined that their GE Magne-Blast 4160 volt circuit breaker overhaul Procedure 32HT-9ZZ38,
"Overhaul of AH-4. 16-250-9H G.E. Hagne-Blast Circuit Breakers,"
would be improved by documenting shim locations.
The licensee issued I
?
C
?
R
?
2546 to document any changes to shim locations in GE 41K v~o t HagneeTast circuit breaker maintenance and overhaul procedures.
The licensee stated that they intended to issue the procedure changes by April 1,- 1994.
The inspector concluded that the l.icensee's GE 4160 volt Hagne-Blast circuit-breaker shim control policy was not in violation of NRC requirements and that the licensee had adequately resolved the technical concern by committing to document changes in shim locations in their associated overhaul procedures.
ll ON-SITE REVIEW OF LICENSEE EVENT REPORTS (LERs)
(90712)
11. 1 Closed LER 50-528 93-02 Revision 0:
Loss of Train B Diesel Generator Com onents Due to a Control Room fire The licensee had determined that a control room fire, concurrent with a loss of offsite power could cause the failure of EDG B.
In addition, the licensee determined that their procedures did not provide sufficient guidance to restore EDG B; therefore, a single fire could adversely affect the ability to achieve and maintain safe shutdown (SSD).
The licensee stated that the cause of the problem was failure of their original
CFR Part 50, Appendix R,
analysis to discover'a design deficiency in the electrical design of the control room isolation transfer switch circuitry for EDG B.
The licensee revised Procedure 41AO-lZZ44, Revision 4, "Control Room Fire,"
Appendix E, to instruct operations personnel to take manual. action to perform the EDG B transfer as one of the first operator actions to take after evacuating the control room and to provide specific restoration instructions to restore EDG B, if the transfer was not accomplished prior to circuit failure.
The licensee staged replacement fuses for circuit restoration.
The licensee studied the time required to restore EDG B and determined that the time required was acceptable for safe shutdown.
In addition, the licensee initiated a
CFR Part 50, Appendix R, reconstitution project to perform an in-depth review of the existing safe shutdown analysis.
The inspector also reviewed Appendix E of Procedure 41AO-lZZ44 and determined that it provided adequate instructions to properly restore EDG B.
The inspector reviewed the latest performance of Procedure 14FT-9FP06, Revision 4,
"Fire Equipment Locker and Emergency Equipment Cabinet Inspection,"
and determined that the required fuses were staged.
Based on these reviews and the licensee's performance of the
CFR Part 50, Appendix R, reconstitution, the inspector considered this LER adequately resolve r t
'
I
)
v I
i
e t,
I
"I I
i
)
I
-27-11.2 Closed LER 50-528 93-05 Revision 0:
Loss of Redundant Trains of SSD E ui ment Due to a Sin le Fire During their 10 CFR Part 50, Appendix R, reconstitution project review, the licensee determined that redundant trains of SSD equipment could be rendered inoperable by a single fire.
Th'is condition was caused by a design which provided fuse isolation between SSD equipment and other, associated equipment on only one leg of 125 volt direct current (dc).
The licensee determined that the associated circuits were not fire protected; therefore, shorts between the nonfused wiring and ground could occur, which could trip a backup circuit breaker supplying power to SSD loads.
The licensee's other train of SSD equipment was also postulated to have failed, since unprotected cables or equipment from this train also existed in the same fire area.
This condition existed for eight pieces of SSD equipment.
The licensee took a number of interim corrective actions.
The licensee established fire watches in the affected areas.
The licensee issued Revision 6 to their Pre-Fire Strategies Hanual to provide a detailed procedure for circuit restoration, in case of circuit failure during a fire.
The licensee staged spare fuses to support equipment restoration and added the fuses to Procedure 14FT-9FP06, Revision 4, "Fire Equipment Locker and Emergency Equipment Cabinet Inspection."
For the equipment affected by a control room fire, the licensee updated Procedure 41A0-1ZZ44, Revision 4,
"Control Room Fire," Appendix H, to provide a detailed procedure for circuit restoration.
As a final corrective action, the licensee committed to initiate a design change to correct the problem.
The inspector reviewed sample records of fire watches, the Pre-Fire Strategies Hanual, Appendix H, of the Control Room Fire Procedure, and the records from the latest performance of Procedure 14FT-9FP06.
The inspector found that the committed fire watches were being performed, that the Pre-Fire Strategies Hanual and Control Room Fire Procedures contained adequate instructions for circuit restoration, and that the required fuses were properly staged.
The licensee stated that they still planned to install a design change as a final corrective action.
The inspector concluded that the licensee's interim and planned final corrective actions were adequate.
IN OFFICE REVIEM OF LERs (90712)
Unit 1:
LER 528/93-09, Revision 1, Hain Steam.Safety Valve and Pressurizer Safety Valve Setpoints Out of Tolerance.
Unit 2:
LER 529/93-02, Revision 2, Hain Steam Safety Valve and Pressurizer Safety Valve Setpoints Out of Toleranc,
I I
'
-28-LER 529/93-05, Revisions 0 and 1, Failure to Demonstrate Operability of A. C.
Offsite Sources Within 1 Hou ~ C,~
(
]
ATTACHMENT 1
Persons Contacted 1. 1 Arizona Public Service Com an
- R. Adney, Plant Manager, Unit 3
- J. Bailey, Assistant Vice President, Nuclear Engineering
- R. Bouquot, Supervisor, guality Assurance Audits
- W. Chapin, Manager, Refueling and Maintenance Services
"J. Dennis, Manager, Operations Standards
- A. Fakhar, Manager, Mechanical Group, Site Technical Support
- R. Flood, Plant Manager, Unit 2 D. Garchow, Director, Site Technical Support
- D. Gouge, Director, Plant Support
- B. Grabo, Supervisor, Nuclear Regulatory Affairs W. Ide, Plant Manager, Unit
- J. Levine, Vice President, Nuclear Production D. Mauldin, Director, Site Haintenance and Modifications
- F. Riedel, Manager, Operations, Unit
K. Roberson, Senior Engineer, Nuclear Regulatory Affairs J. Scott, Assistant Plant Manager, Unit 3 C.
Seaman, Director, guality Assurance and Control
- J. Steward, Manager, Radiation Protection
- J. Terry, General Manager, Nuclear Records Management
- S. Troisi, Manager, Site Technical Support
- P. Wiley, Manager, Operations, Unit 2 1.2'thers
- J. Draper, Site Representative, Southern California Edison
- R. Henry, Site Representative, Salt River Project
- F. Gowers, Site Representative, El Paso Electric 1.3 NRC Personnel
- G. Johnston, Licensee Examiner
- J. Lynch, Contract Inspector, Office of Nuclear Reactor Regulation
- Denotes personnel in attendance at the Exit meeting held with the NRC resident inspectors on March 31, 1994.
EXIT MEETING An exit meeting was conducted on March 31, 1994.
During this meeting, the inspectors summarized the scope and findings of the report.
The licensee acknowledged the inspection findings documented in this report.
The licensee did not identify as proprietary any information provided to, or reviewed by, the inspector c.. y l
I I
f
!
t