IR 05000528/1998009
| ML17313A788 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 02/08/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17313A784 | List: |
| References | |
| 50-528-98-09, 50-528-98-9, 50-529-98-09, 50-529-98-9, 50-530-98-09, 50-530-98-9, NUDOCS 9902120215 | |
| Download: ML17313A788 (47) | |
Text
ENCLOSURE 2 U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Docket Nos.:
50-528 50-529 50-530 License Nos.: NPF-41 NPF-51 NPF-74 Report No.:
50-528/98-09 50-529/98-09
'50-530/98-09 Licensee:
Facility:
Location:
Dates:
Inspectors:
Arizona Public Service Company Palo Verde Nuclear Generating Station, Units 1, 2, and 3 5951 S. Wintersburg Road Tonopah, Arizona November 15 through December 26, 1998 J. Moorman, Senior Resident Inspector D. Carter, Resident Inspector D. Corporandy, Resident Inspector N. Salgado, Resident Inspector Approved By: P. Harrell, Chief, Project Branch D Attachment:
Supplemental Information 9902i20215 9'90208 PDR ADQCK 05000528
l
EXECUTIVE SUMMARY Palo Verde Nuclear Generating Station, Units 1, 2, and 3 NRC Inspection Report 50-528/98-09; 50-529/98-09; 50-530/98-09
~Oeratiens Valve 3JSIBHV690 (shutdown cooling (SDC) system warmup line isolation valve) was found by the inspectors to be approximately 10 percent open with Unit 3 in the SDC mode of operation.
This allowed a small amount of coolant to bypass the core; however, sufficient flowwas available to the core. This resulted from an operator not placing SDC Train B in service, during the Unit 3 refueling outage, in accordance with the applicable procedure.
This is a repeat violation of Technical Specification (TS) 5.4.1 for failure to followa procedure (Section 01.1).
Based on a review of a sample of seven clearances installed on the auxiliary feedwater and containment spray systems in Unit 2, it was noted that they were properly prepared, authorized, implemented, and restored (Section 01.2).
Maintenance Craftspersons used approved procedures to perform routine maintenance activities in a safety conscious manner.
Good work and foreign material control practices were observed (Section M1.1).
Troubleshooting of the Unit 1 Train A balance-of-plant engineered safety features actuation system and preparations in advance of the troubleshooting were accomplished in a safe manner with good coordination between the technicians and Operations personnel (Section M1.2).
Qualified personnel used approved procedures to conduct surveillance activities in a safety conscious manner (Section M1.3).
Rupture of the low pressure portion of the Unit 1 postaccident sampling system (PASS)
on July 16, 1998, resulted from a failure to take timely and appropriate actions to correct four maintenance rule repetitive functional failures that occurred between 1993 and 1998, a violation of 10 CFR 50.65(a)1.
Inadequate evaluation of the Unit 1 PASS history prior to July 10, 1996, for maintenance rule repeat functional failures resulted in the failure to monitor and establish goals for the system for 6 months after implementation of the maintenance rule. The licensee identified this issue as a significant Quality Assurance program breakdown.
This was licensee-identified and is a noncited violation of 10 CFR 50.65(a)2 (Section M1.4).
The observed material condition of the three units was satisfactory (Section M2.1).
The mechanical maintenance investigation and repair of the Unit 2 Emergency Diesel Generator B outboard generator bearing was good. Mechanics demonstrated good adherence to the troubleshooting action plan and properly requested the action plan be
l I
(
t
-2-modified and approved prior to modifications of the action plan being performed.
Evaluation of the root cause for failure and transportability of the issue to the other onsite emergency diesel generators was good (Section M4.2).
The licensee's "1997 Maintenance Rule Periodic Assessment" satisfied the requirements of 10 CFR 50.65 paragraph (a)(3). Corrective actions for program weaknesses identified in the assessment were being tracked by the licensee's Commitment Action Tracking System.
Identified weaknesses and the licensee's conclusion that the program was being effectively administered were consistent with prior NRC findings (Section M7.1).
~En ineerin
~
The licensee effectively implemented the cold weather protection procedure to protect safety-related components and piping that are susceptible to freezing temperatures (Section E3.1).
Plant Su ort
~
A good process for monitoring and removing boron buildup on safety-related components was implemented by the licensee. (Section R4.1)
Re ort Details Summa of Plant Status Units 1, 2, and 3 operated at essentially 100 percent power for the duration of this inspection period.
I. 0 eratlons
Conduct of Operations 01.1 Failure to Follow Procedure Resulted in Valve 3JSIBHV690 Remainin 0 en While in the SDC Mode of 0 eration Unit 3 a.
Ins ection Sco e 71707 On October 21, 1998, while independently verifying the SDC system lineup, the inspectors observed that Valve 3JSIBHV690 (SDC system warmup line isolation valve),
was approximately 10 percent from the full-closed position. The inspectors reported the condition to the control room and reviewed the circumstances around the mispositlonirig of the valve.
b.
Observations and Findin s On October 21, Unit 3 was in Mode.5 with reactor coolant loops filled and SDC Train B in service.
During a tour of the auxiliary building, the inspectors observed that Valve 3JSIBHV690 was approximately 10 percent from the fully-closed position, as indicated by the calibrated local valve position indicator. The valve was required to be 100 percent closed.
The inspectors informed the shift manager of the condition, who directed an operator to close the valve. Indicated SDC flowon the control board at the time was approximately 4500 gpm, well above the Technical Specifications (TS)
required value of 3780 gpm. The inspectors reviewed SDC flow trend data from the emergency response facilitydata acquisition and display system and verified that the SDC flowdid not change noticeably after the valve was fullyclosed.
Condition Report/Disposition Request (CRDR) 3-8-0337 was initiated to document the discovery of the mispositioned valve.
NRC Inspection Report 50-528,529,530/97-16 documented a violation of TS 3.4.1.4.1,
"Cold Shutdown - Loops Filled," which occurred on October 6, 1997, in Unit 2. Less than required SDC flow existed for 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> due to the mispositioning of Valve 2JSIBHV690. Corrective actions for this violation were documented in Licensee Event Report (LER) 50-529/97-07 and CRDR 2-7-0383.
Corrective actions included the installation of a calibrated local valve position indicator, inclusion of additional steps in applicable procedures to require local verification of the position of Valves JSIBHV690 and 691 after operation, and industry events training for Operations personnel.
Additionally, the position of Valves JSIBHV690 and -691 was required to be checked and logged in accordance with Procedure 40ST-9ZZ1 9, "Modes 5 8 6 Shiftly Surveillance Logs," Revision 20, once per shift by auxiliary operators (AO) while an SDC train was in operatio t h
e
-2-The inspectors reviewed CRDR 3-8-0337 and determined that, on October 21, 1998, SDC Train B was placed in service in accordance with Section 9 of Procedure 40OP-9SI01, "Shutdown Cooling Initiation," Revision 9. The CRDR included a personnel statement made by the reactor operator who performed the evolution. The operator stated that local verification of valve position was not conducted as required by Procedure 40OP-9SI01.
The operator attributed this omission to his not using placekeepers when performing the procedure.
This failure to followa procedure is a repeat violation of TS 5.4.1 (50-530/9809-01).
The licensee's investigation of the event determined that an additional opportunity existed to close Valve 3JSIBHV690.
In a personnel statement, an AO stated that, during performance of Procedure 40ST-9ZZ19, he observed the valve and thought it to be slightly open; however, the AO did not inform the control room of the possible incorrect position. The AO stated that he had experience with similar type valve position indicators that were not calibrated and not as accurate as the ones installed on Valves JSIBHV690 and 691. These noncalibrated indicators could indicate a valve was slightly open although the valve was fullyshut. The AO also stated that he was not familiar with previous problems verifying the position of the SDC system warmup line isolation valves.
CRDR 3-8-0337 provided corrective actions for the most recent event. The interim corrective action was to issue a night order that required the valve handswitch to be held in the closed position for an additional 5 seconds by an independent operator when closing the valve: Long-term corrective actions included making revisions to Procedures 40OP-9SI01, 40ST-9ZZ1 9, and 40OP-9SI02, "Recovery From Shutdown Cooling to Normal Operating Lineup," Revision 23. Additionally, both operators were coached on procedure adherence and the event willbe included in industry events training, with an emphasis on how to read local valve stem indicators. The inspectors considered the corrective actions to be adequate to prevent recurrence.
Conclusions Valve 3JSIBHV690 (SDC. warmup line isolation valve) was found by the inspectors to be approximately 10 percent open with Unit 3 in the SDC mode of operation.
This allowed a small amount of coolant to bypass the core; however, sufficient flowwas available to the core. This resulted from an operator not placing shutdown cooling Train B in service, during the Unit 3 refueling outage, in accordance with the applicable procedure.
This is a repeat violation of TS 5.4.1 for failure to followa procedure.
Auxilia Feedwater S stem and Containment S ra CS S stem Clearance Review Unit 2 Ins ection Sco e 71707 62707 The inspectors reviewed several auxiliary feedwater and CS system clearances to assess the implementation, restoration, and completeness of the clearance package b.
Observations and Findin s The inspectors independently verified, on a sampling basis, that the tagged components associated with the following clearances were in the required positions with the appropriate tags in place:
98-01022
"Inspect Lubrication; Lubricate Valve Stem; Functionally Test Motor Operator for 2JAFBHV0031" 98-01023
"Inspect Lubrication; Lubricate Valve Stem; Functionally Test Motor Operator for 2JAFBHV0034" 98-01024 inspect Lubrication; Lubricate Valve Stem; Functionally Test Motor Operator for 2JAFBHV0035" 98-01025
"Check Pump Alignment and Mechanical Center Of The Motor Per The Applicable Sections Of 31 MT-9AF01" 98-01040
"Inspect Lubrication; Lubricate Valve Stem; Functionally Test Motor Operator for 2JSIBHV0689" 98-01039
"Inspect Lubrication; Lubricate Valve Stem; Functionally Test Motor Operator for 2JSIBUV0665" 98-01038
"Lubrication Program - Sample and Change Oil/Sample Oil in Containment Spray Pump Motor Upper Bearing, Sample and Change Oil in Lower Bearing, Use Turbo 46 Oil" The inspectors also observed AOs while they were placing tags on components, as required by the respective clearance tag assignment sheets.
The AOs performed independent verification in accordance with plant procedures.
During the racking down of the circuit breaker associated with the CS pump, the AOs adhered to the instructions in Procedure 42OP-2PB01, "4.16kV Class 1 E Power (PB)," Appendix D, Revision 16.
For the clearances associated with the CS system, the inspectors verified that technicians performing work in the field had accepted the clearances in the control room prior to starting the work on their respective work orders (WO).
The inspectors also verified, in part, that the equipment was restored to service appropriately and that the clearance tags were removed.
c.
Conclusions Based on a review of a sample of seven clearances installed on the auxiliary feedwater and containment spray systems in Unit 2, it was noted that they were properly prepared, authorized, implemented, and restore P l
f
Miscellaneous Operations Issues
'08.1 Closed LER 50-530/9801-00:
TS 3.0.3 Entry Due To Safety injection Flow Instruments Being Removed From Service.
This item was discussed and dispositioned as a violation in NRC Inspection Report 50-528;529;530/98-06.
No new issues were revealed during review of this LER.
II. Maintenance M1 Conduct of Maintenance M1.1 General Comments on Maintenance Activities a.
Ins ection Sco e 62707 The inspectors observed all or portions of the following work activities:
854722 846388 824101
"Diesel Generator - A Lube Oil Circ Pump Seal Replacement" (Unit 3)
"Disassemble/Inspect AirStart Check Valves" (Unit 3)
"2MSIBP03 - Sample and Change Oil In Motor Bearings" (Unit 2)
32MT-9ZZ52 "Preventative Maintenance Procedure Battery Charger,'evision
(Unit 1)
b.
Observations and Findin s The inspectors found the work performed under these activities to be properly performed.
Allwork observed was performed with the work package present and in active use. Work and foreign material exclusion practices observed were good.
Technicians were experienced and knowledgeable of their assigned tasks.
C.
Conclusions Craftspersons used approved procedures to perform routine maintenance activities in a safety conscious manner.
Good work and foreign material control practices were observed.
M1.2 Balance-of-Plant En ineered Safe Features Actuation S stem BOP ESFAS Troubleshootin Unit 1 a.
Ins ection Sco e 62707 The inspectors observed portions of the performance of WO 844218, "Troubleshooting of Train A Balance of Plant Engineered Safety Features Actuation System."
I
-5-Observations and Findin s The BOP ESFAS provides initiating signals to engineered safety feature components requiring automatic actuation.
Initiating signals are actuated whenever monitored safety parameters exceed their setpoint values, the point at which protective actions are required. Another significant feature provided by BOP ESFAS is the sequencing of safety loads, as required, to prevent instability of the Class 1E 4.16-kV buses.
BOP ESFAS is periodically tested to verify its performance.
Following testing, the BOP ESFAS load sequencer is reset to return the system to its normal status.
On occasion, the Unit 1 Train A load sequencer has locked up when attempting to reset it after testing. This flaw would not prevent BOP ESFAS from performing, as designed, once it.
has been reset to its normal status.
In preparation for troubleshooting, the operators had to deenergize safety equipment that could potentially be actuated during the BOP ESFAS troubleshooting activities. The inspectors observed portions of this work by Operations personnel.
Work observed by the inspectors was successfully accomplished, with good coordination between the control room operators and the AO in the field.
The inspectors also observed portions of the troubleshooting activities performed by the Instrument and Control (l&C)engineers and technicians.
I&C had developed a plan to systematically conduct their troubleshooting efforts. Precautions discussed in the sensitive issues briefing preceding the activities were closely followed. The sequencer performed well during the troubleshooting, and l8C was unable to reproduce a sequencer lockup despite numerous actuations and resets of the system and several power supply manipulations.
However, I&C located and tightened several slightly loose connections in the system, replaced two suspect resistors and diodes, and obtained numerous measurements of system parameters while performing actuations, resets and power supply breaker manipulations.
I&C intended that the data collected during these troubleshooting activities would be valuable for establishing nominal values.
These nominal values were compared with those taken for Unit 3, and it was noted that the Unit 1 values were consistent.
In addition, l8C found that intermittent connections-to relays that are driven directly from the sequencer can cause sequencer lockups. This was accomplished by intentionally loosening connections for test purposes.
Conclusions Troubleshooting of the Unit 1 Train A balance-of-plant engineered safety features actuation system and preparations in advance of the troubleshooting were accomplished in a safe manner, with good coordination between the technicians and Operations personne l I
'I
-6-M1.3 General Comments on Surveillance Activities a.
Ins ection Sco e 61726 The inspectors observed all or portions of the following surveillance activities:
"Diesel Generator B Test," Revision 8 (Unit 3)
"Essential Spray Pond Pumps - Inservice Test," Revision 11 (Unit 2)
"Containment Spray Pumps and Check Valves - Inservice Test,"
Revision 5 (Unit 2)
b.
Observations and Findin s The inspectors found that knowledgeable personnel performed these surveillances satisfactorily, as specified by applicable procedures.
c.
Conclusions
.
Qualified personnel used approved procedures to conduct surveillance activities in a safety conscious manner.
M1.4 Reactor Coolant S stem RCS Leak From the PASS Unit 1 a.
Ins ection Sco e 37551 On July 16, 1998, the low pressure portion of the PASS located in the west mechanical penetration room (WMPR) on the 70-foot level of the auxiliary building was overpressurized, which resulted in the loss of RCS inventory and the contamination of the room. The inspectors reviewed operator logs and the circumstances surrounding the event and conducted interviews with licensee personnel.
The inspectors also reviewed CRDRs associated with previous Unit 1 PASS overpressurization events.
Observation and Findin s At 9:30 p.m. on July 16, 1998, a chemistry technician aligned PASS to sample the RCS hot leg in accordance with Procedure 74ST-9SS04, "PASS Functional Test," Revision 6.
This procedure directed the technician to perform Section 6.3 of Procedure 74OP-1SS02, "Operation of the Post Accident Sampling System," Revision 25, to draw the sample.
The technician lined up the RCS sample to the equipment drain tank (EDT)
and commenced a 30-minute purge of the system.
The technician left the PASS control panel to perform other duties while waiting for the system to purge.
At 10:16 p.m., the control room received an alarm on Panel RU-9, "AuxiliaryBuilding Lower Level Ventilation Exhaust Monitor." The gas channel on this panel indicated a slow increasing trend. The control room dispatched an AO to evaluate conditions in the
-7-auxiliary building. At 10:23 p.m., the control room received an alarm on RU-8, "Auxiliary Building Upper Level Ventilation Exhaust Monitor." The iodide channel on this panel indicated a slow increasing trend. The control'room entered Procedure 40AO-9ZZ02,
"Excessive RCS Leakrate," Revision 3, when RCS leakrate increased.
At 10:27 p.m.,
the control room received a fire alarm in the auxiliary building WMPR. The control room dispatched a second AO to investigate this situation. The second AO reported that water and steam were present in the room. At 10:32 p.m., the control'room isolated letdown in response to an increasing RCS leak rate. Control room operators recognized that chemistry was performing PASS sampling and directed that PASS sampling be secured and the PASS be isolated.
At 10:37 p.m., the chemistry technician informed the control room that the PASS was isolated.
The AO near the WMPR reported that the water and steam leakage had stopped.
The PASS was quarantined and the licensee initiated CRDR 1-8-0397 to document and initiate an investigation of the event. The maximum flow rate from the RCS, as indicated by the plant computer, was 5 gpm.
Operations personnel response to the RCS leak was good.
The licensee developed an action plan to determine the cause of the system rupture and performed repairs to the system.
The significant investigation report detailed the sequence of events, failure mode investigation, root cause determination, and corrective actions to prevent recurrence.
The CRDR root cause determination identified four possible causes of the overpressurization event:
(1) isolation of the discharge flow path, (2) inadequate overpressurization protection, (3) high pressure boundary valve leakage, and (4) inadvertent operator action. The CRDR identified the most probable cause was the inadvertent opening of Valve HV17A, which bypassed the system pressure reducing element, causing the low pressure piping to be exposed to full RCS pressure.
The inspectors reviewed the licensee's initial corrective actions to repair the failed components and restore the PASS to operation.
The inspectors noted that the actions were adequate.
The inspectors interviewed the chemistry technician and the PASS system engineer and reviewed past CRDRs and engineering evaluation requests (EER) associated with overpressurization events and failures of system components located in the low pressure portion of the PASS. The inspectors determined that the license had experienced approximately 12 overpressurization events associated with the Unit 1 PASS between August 1985 and July 1998. Units 2 and 3 have different PASS designs and are not as susceptible to the type of overpressurization events experienced in Unit 1.
EER 89-SS-004 E
EER 89-SS-004, dated February 24, 1989, described an overpressurization event of the Unit 1 PASS on February 8, 1989. The licensee's investigation indicated that the most likely cause of the event was the misoperation of pressure reducing element bypass Valve HV-17A. Inadvertent opening of this valve allowed high RCS pressure to bypass the pressure reducing element, resulting in overpressurization of the downstream piping and the failure of pressure relief Valve PSV-19 and pressure control Valve PCV-19. The licensee's recommended corrective action was to provide overpressure protection to the
f
-8-low pressure portions of the PASS. This corrective action was not instituted prior to the next overpressurization event on December 28, 1989 (see EER 90-SS-002 discussion below).
EER 90-SS-002 EER 90-SS-002, dated December 28, 1989, described leakage past PASS booster pump discharge check Valve 2PSSNV874, which allowed back leakage to overpressurize the PASS booster pump and cause the seal to fail. The licensee initiated Plant Change Request (PCR) 90-13-SS-001, on February 15, 1990, to provide overpressure protection for the low pressure portions of the PASS.
PCR 90-13-SS-001 was approved on April3, 1990. Subsequently, on February 22, 1994, the licensee reviewed this PCR for cancellation and canceled it on November 21, 1994. The licensee canceled the PCR corrective action 20 days after another PASS overpressurization event occurred on February 4, 1994 (see CRDR 1-4-0049 discussion below).
EER 90-SS-016 EER 90-SS-016, dated April 24, 1990, described an overpressurization event of the low pressure portion of the Unit 1 PASS due to an improper sample return flow path to either the reactor coolant drain tank (RDT) or EDT. With neither of the return flowpaths lined up properly, the pressure downstream of the pressure reducing element increased and overpressurized the low pressure portion of the PASS.
In addition, the pressure relief valve designed to protect the low pressure portion of the PASS relieves inside of the downstream RDT or EDT isolation Valves HV-4 and HV-5. With these valves shut, the relief valve could not protect the low pressure portion of the PASS from overpressurization.
The licensees corrective action revised the sample procedure to require signature verification that the correct valve line up existed to provide a sample return flow path to the RDT or EDT, prior to directing flowto one of the tanks.
CRDR 1-4-0049 CRDR 1-4-0049, dated February 4, 1994, described an RCS leak from the PASS. The leak resulted when back pressure control Valve PCV-19 failed. The licensee's evaluation suggested that the reason for the failure of Valve PCV-19 was overpressurization of the low pressure portion of the PASS, where this component is located.
The,CRDR documented that similar occurrences of this type of failure were noted in the Unit 1 PASS history. The CRDR also indicated that these failures are typically caused by blockages of the return path isolation valves to the RDT or EDT.
This allowed equalization of pressure across the pressure reducing element and subsequent failure of the tow pressure PASS components.
The CRDR also described a high flow rate condition through the PASS that could have come from pressure reducing element bypass Valve HV-17A being open or leaking past its seat during RCS hot leg recirculation. The high flow rate could cause high temperatu're RCS coolant to pass through the PASS system and exceed the design temperature (180 F) of the tow pressure portion of the system.
High temperature coolant caused damage to some downstream valves with Teflon valve seats, which can cause them to stick shu l I
-9-Repairs were made to the system by replacing Valve PCV-19, pressure relief Valve PSV-19, and flow Switch FS-2. The licensee's investigation concluded that the cause of the system overpressurization was indeterminate.
No further investigation was recommended and the CRDR was closed after review by Engineering.
This PASS overpressurization event was very similar to the most recent July 16, 1998, event.
CRDR 9-6-0666 CRDR 9-6-0666, dated June 26, 1996, described two occasions in which Valve HV-19 had a deformed valve plug and that the valve was stuck shut due to overheating.
This is the same type of problem as described in CRDR 1-4-0049, disucssed above.
The CRDR identified that WOs 737166 and 761471, completed on December 15, 1995, and June 25, 1996, were written to document the valve failures. Neither of the two failures were documented on a CRDR. The evaluation in this CRDR indicated that there was a third instance of Valve HV-19 being damaged due to excessive heat, as documented in WO 733693, dated November 17, 1995.
In all three instances, Valve HV-19 required replacement.
This was the third time in a 28-month period that Valve HV-19 was replaced due to a deformed valve plug caused by overheating.
Valve HV-19 exceeded 2 failures in 10 demands for the first time in December 1995 and again in June 1996.
The evaluation of this CRDR also indicated that there was no method for measuring temperature of the sample before or after the sample cooler. The CRDR evaluation stated, in part, that an evaluation of Valve HV-19 failures, and changes to allow Valve HV-19 to withstand higher temperatures, would be performed.
The licensee's corrective action associated with this CRDR was to evaluate the need for a design change.
This design change corrective action was assigned as a Priority Level 3 action. The Priority Level 3 corrective action was later changed to a Priority Level 4 action.
Priority Level 4 actions are not considered to be corrective actions; therefore, a corrective action plan was not documented on the CRDR when the system was placed in MR Category a(1).
The licensee answered the Priority Level 4 action item in Memorandum 203-02085-TRA, dated December 4, 1996. This memorandum recommended that a design change be developed to relocate the toggle switch for Valve HV-17A. Engineering concluded that the overpressurization and overheating issues associated with the low pressure portion of the PASS were human factors issues.
The toggle switch for activating Valve HV-17A is located directly below (within 2 inches) the toggle switch for pressure reducing element inlet Valve HV-16, and there is a high probability of inadvertently brushing the Valve HV-17Aswitch. The licensee initiated Deficiency Work Order (DFWO) 778402 to relocate the toggle switch for Valve HV-17A away from the toggle switch for Valve HV-16 and to install a cover box over Switch HS-17A to prevent inadvertent actuation.
The licensee's MR Expert Panel reviewed the recommendations and set a completion date goal of March 1, 1998, for performing DFWO 778402.
In February 1998, the licensee postponed the completion date of DFWO 778402 until May 1998.
Design Engineering did not complete the engineering to perform the modification in time for the Unit 1 refueling outage scheduled for March 199 t I
t
l
-10-Prior to the May 1998 modification, a new design engineer was assigned to the PASS and his review of DFWO 778402 resulted in it being canceled.
The moving of the switch and installation of a cover was found not to be cost effective. Design Engineering generated Material Engineering Evaluation 02390, which allowed the type of toggle switch for Valve HV-17A to be changed from a simple toggle-type switch to a positive-action type switch. This new type of switch was proposed to prevent the inadvertent operation of Valve HV-17Aand was less costly to implement. The licensee initiated Work Request 945456 to replace the toggle switch for Valve HV-17Aand scheduled the modification for September 29, 1998, under WO 849723:
PASS Performance Evaluation The licensee included the PASS within the scope of the MR because the system provides a contribution toward meeting the Maintenance of Vital Auxiliaries safety function and its use in the emergency operating procedures.
The licensee's MR performance criteria for the PASS were: a 5 percent RCS unavailability and reliabilityof greater than or equal to 95 percent.
The system must be capable of obtaining an RCS sample which meets acceptance criteria when compared with routine nuclear sampling analytical results.
The performance would not be acceptable if there were 4 failures in 40 demands and 2 failures in 10 demands.
A review of performance for advance trends willbe performed if there are 2 failures in 10 demands.
The licensee placed the PASS into MR monitoring category (a)(2) when the MR went into effect on July 10, 1996.
MR Repetitive Functional Failures (MRRFF) of Valve HV-19 occurred in November and December 1995, and in June 1996, as described in CRDR 9-6-0666. These failures were documented on WOs and were not recognized as MRRFFs by the licensee during their initial data assessment conducted prior to the implementation of the MR. Data collection for repetitive failure determination started from June 1993. CRDR 9-6-0666 identified the June 1996 failure as a repeat of the December 1995 failure. 'The system engineer had discovered that Valve HV-19 experienced three MRRFFs, while reviewing quarterly Failure Data Trending reports in response to CRDR 9-6-0666.
The PASS was not placed in monitoring category (a)(1) to address these failures until January 1997, 6 months after the implementation of the MR.
a An independent review of this event and previous events by the inspectors identified that the licensee failed to monitor the performance or condition of the PASS against licensee established goals. As a result, insufficient monitoring failed to identify that the PASS was not capable of fulfillingits intended function. The licensee failed to evaluate and place the PASS in MR monitoring category a(1) until January 1997, after Valve HV-19 had experienced three MRRFFs.
Failure to perform adequate monitoring of established performance criteria is a violation of 10 CFR 50.65(a)2.
However, this violation is considered to be noncited, consistent with the requirements of Section VII.B.1 of the Enforcement Policy. Specifically, the violation was nonrepetitive and self-identified, it was not willful,actions taken as a result of a previous violation should not have corrected this problem, and appropriate corrective actions were completed or are incorporated into the licensee's corrective action program (50-528/9809-02).
tl
-11-On July 24, 1998, the licensee initiated CRDR 9-8-1180, after the inspectors questioned the system engineer on the status of corrective actions taken for the June 26, 1996, event (as documented in CRDR 9-6-0666). This CRDR described that ineffective and untimely corrective actions of the June 26, 1996, event allowed an MRRFF of Valve HV-19 on July 16, 1998. The CRDR also stated, in part, that, if the toggle switch for Valve HV-17A been moved and covered or replaced with the a new positive-action type, as recommended in CRDR 9-6-0666 corrective actions, this failure would most likely have not occurred.
The licensee failed to take adequate corrective actions to prevent recurrence of the Unit 1 PASS overpressurization events.
Several proposed corrective actions had been initiated over the years to correct the PASS problems.
However, the licensee either continued trending the conditions or postponed or canceled several corrective action recommendations that could have prevented recurrence of system ruptures.
The failure to implement actions to prevent recurring ruptures in the PASS is a violation of 10 CFR 50.65(a)1 (50-528/9809-03).
'he licensee initiated CRDR 9-8-1212 on July 31, 1998, which identified several MR programmatic problems associated with the PASS. This CRDR identified that performance monitoring guidance for the MR is provided by Procedure 30DP-OMR01,
"Maintenance Rule," Revision 3, which stated that functional failure determinations are handled and resolved using the licensee's CRDR process.
The licensee identified several examples where CRDRs were not written to document equipment failures that occurred during the performance of PASS surveillance tests.
Procedure 30DP-OMR01 also stated that, when a failure occurred on an MR system, a Potentially Significant CRDR willbe submitted to the responsible system engineer for a MRRFF determination.
The licensee identified several examples of CRDRs that identified PASS equipment failures that were not classified as potentially significant and did not request a MRRFF determination.
The licensee also identified that corrective actions and followup of MR (a)(1) monitoring goals by the MR Expert Panel were not adequate or timely in several instances.
The licensee replaced the toggle switch for Valve HV-17Aon September 29, 1998, with a new design, as part of the corrective action for CRDR 1-8-0397. The licensee's planned corrective actions were to: (1) revise the PASS MR System basis document to clarify monitored system functions and methods; (2) develop and document a process to ensure PASS equipment failures are promptly and appropriately documented on CRDRs; (3) review all identified PASS MRRFFs to ensure failures have been addressed by completed or planned corrective actions; (4) review all PASS components currently in Category (a)(1) to ensure that corrective actions have been developed and are being tracked by the CRDR process; and (5) conduct an assessment of the transportability of these causes, identified in this CRDR evaluation, to other systems, structures, and components within the scope of the MR. These corrective actions were scheduled for completion by February 1999.
The licensee initiated CRDR 9-8-Q239 to address the root cause of continued inconsistent PASS system performance and untimely and ineffective corrective action to previous events that represented a significant Quality Assurance program breakdown.
The licensee closed CRDR 9-8-1180 and incorporated its concerns into CRDR 9-8-Q239.
Planned corrective actions included incorporation of design and t
l I
t
-12-hardware modifications of the PASS into the unit's work schedule to prevent future overpressurization events.
These modifications were scheduled for completion by February 1999.
In addition, discussions of PASS untimely and ineffective corrective actions were to be included in System Engineering forums and in maintenance and chemistry industry events training. These corrective actions were considered to be appropriate by the inspectors.
c.
Conclusions Rupture of the low pressure portion of the Unit 1 postaccident sampling system on July 16, 1998, resulted from a failure to take timely and appropriate actions to correct four maintenance rule repetitive functional failures that occurred between 1993 and 1998, a violation of 10 CFR 50.65(a)1.
Inadequate evaluation of the Unit 1 postaccident sampling system history prior to July 10, 1996, for maintenance rule repeat functional failures resulted in the failure to monitor and establish goals for the system for 6 months after implementation of the maintenance rule. The licensee identified this issue as a significant Quality Assurance program breakdown.
This was licensee-identified and is a noncited violation of 10 CFR 50.65(a)2.
M2 Maintenance and Material Condition of Facilities and Equipment M2.1 Review of Material Condition Durin Plant Tours Units 1 2 and 3 a.
Ins ection Sco e 62707 During this inspection period, routine tours of all units were conducted to evaluate plant material condition.
b.
Observations and Findin s Based on the observations made by the inspectors, no major observable material condition deficiencies were identified. Minor deficiencies, brought to the attention of the licensee, were documented with work requests.
c.
Conclusions The observed material condition of the three units was satisfactory.
M4.2 Misali ned Emer enc Diesel Generator EDG B Outboard Generator Bearin Unit 2 a.
Ins ection Sco e 62707 On Decelnber 10, 1998, during planned corrective maintenance on the EDG B outboard bearing, the licensee identified that the bearing housing had rotated from its correct alignment. The inspectors reviewed the licensee's investigation and evaluation of this conditio Observations and Findin s During the performance of corrective maintenance to replace temperature trip Valve DGN-TV-0260 on the outboard generator bearing under WO 863989, a mechanic experienced difficultyremoving the temperature probe from the journal bearing housing and identified that the temperature probe was bent. The mechanic noticed that the holes through the bearing pedestal casing, bearing assembly, and bearing, which the temperature probe passed through, did not line up due to the bearing assembly having rotated within its housing.
The mechanic stopped work and informed management of the problem. The journal bearing was quarantined pending the development of a troubleshooting game plan. The licensee informed the inspectors that the scope of work on the EDG would be increased to dismantle and inspect the journal bearing for possible damage.
The inspectors reviewed the prejob briefing and troubleshooting action plan, noted that the plan was detailed, and provided the appropriate information. After reviewing the bearing technical manual and installation instructions, the licensee hypothesized that the bearing plunger screw may have become loose or damaged, which allowed the bearing assembly to rotate. The plunger screw is a holddown screw on the top of the bearing to hold the bearing assembly in place within the housing. The licensee performed bearing oil sample analyses, as part of the root cause of failure determination, and compared it to historical bearing performance data and oil sample results.
No abnormalities were noted.
The mechanics removed the upper bearing housing and observed that the bearing assembly had rotated approximately 3 degrees in the direction of shaft rotation. At this point, as part of the root cause determination, the system engineer proposed that the bearing assembly be repositioned to its correct alignment and reassembled to determine if the plunger screw had been tightened to its proper specification. The mechanics requested that the changes to the troubleshooting plan, for reassembly of the bearing, be documented prior to work being performed.
By observing the reassembled condition of the bearing, the licensee was able to determine that the plunger screw had been approximately 3/4 of a turn loose, which allowed the bearing assembly to rotate within its housing.
The action plan was modified and approved by engineering prior to the additional work being performed.
The licensee dismantled the bearing assembly and inspected the bearing for damage.
No damage or abnormal wear was noted.
The bearing was reassembled and the plunger screw was torqued to its required value.
During reassembly, the mechanics demonstrated good foreign material exclusion practices.
The licensee initiated CRDR 2-8-0298 to document the issue and to determine the transportability of the condition to the site's other five EDGs. The licensee determined from the root cause of the failure investigation and bearing inspection that the EDG bearing had been in this condition since the first EDG run, prior to licensing. While in this condition, the EDG successfully supported all TS testing requirements and actual emergency run demands and operated in excess of 1500 hours0.0174 days <br />0.417 hours <br />0.00248 weeks <br />5.7075e-4 months <br />.
The journal bearing lube oil sample analysis results were satisfactory, with no evidence of bearing
I
I f
l t
-14-degradation.
The licensee determined that this issue was not an immediate transportability concern based on the root cause investigation results.
The licensee had dismantled and inspected both of Unit 1 EDG generator journal bearings and the Unit 2 EDG A bearing since initial licensing. The licensee willvisually inspect the Unit 3 EDG bearings during the next regularly scheduled work window. The inspectors considered these actions to be appropriate.
Conclusions The mechanical maintenance investigation and repair of the Unit 2 Emergency Diesel Generator B outboard generator bearing was good.
Mechanics demonstrated good adherence to the troubleshooting action plan and properly requested the action plan be modified and approved prior to modifications of the action plan being performed.
Evaluation of the root cause for failure and transportability of the issue to the other
. onsite emergency diesel generators was good.
M7 M7.1 Quality Assurance in Maintenance Activities Review of 1997 Maintenance Rule MR Periodic Assessment Units 1 2 and 3 Ins ection Sco e 62707 The inspectors reviewed the licensee's "1997 Maintenance Rule Periodic Assessment, February 1997 through February 1998" and associated documents developed by the licensee to track identified program deficiencies requiring corrective actions.
The licensee's assessment was performed to satisfy the requirements of 10 CFR 50.65, paragraph (a)(3).
b.
Observations and Findin s The licensee's assessment concluded that structures, systems, and components within the scope of the MR were being effectively monitored in accordance with the requirements of the rule. The assessment stated that an appropriate balance of reliability and unavailability was demonstrated for high-risk significant systems within the scope of the rule. The review also noted that adequate assessment, monitoring, and control of risk when performing maintenance was also demonstrated.
The assessment also identified a number of minor MR issues.
These issues included problems with program documentation that was either incomplete or needed clarification and timely issuance of failure data trending reports.
Part of the problem was that the corrective actions were not formally documented or tracked using the licensee's corrective action program. The inspectors verified that the licensee had initiated CRDR 98-1583 to formally document corrective actions for the identified issues, establish due dates for their completion, and entered this information into the Commitment Action Tracking System.
The inspectors also reviewed CRDR 98-1225, which identified engineers who had not completed a timely review of failure data trendin The inspectors reviewed the MR program issues identified in NRC inspection reports for the period of February 1997 through February 1998 and observed the licensee's "1997 Maintenance Rule Periodic Assessment" to be consistent with NRC findings.
C.
Conclusions The licensee's "1997 Maintenance Rule Periodic Assessment" satisfied the requirements of 10 CFR 50.65, paragraph (a)(3). Corrective actions for program weaknesses identified in the assessment were being tracked by the licensee s Commitment Action Tracking System.
Identified weaknesses and the licensee's conclusion that the program was being effectively administered were consistent with prior NRC findings.
III. En ineerin E3 Engineering Procedures and Oocumentation E3.1 Cold Weather Protection Pro ram Units 1 2 and 3 Ins ection Sco e 37551 71707 t
The inspectors reviewed the licensee's cold weather protection program and conducted walkdowns of safety-related systems, structures, and components that are susceptible to freezing temperatures.
The inspectors also reviewed the Updated Final Safety Analysis Report and piping and instrumentation drawings and interviewed the essential spray pond (ESP) system design engineer.
b.
Observations and Findin s Procedure 40OP-9ZZ17, "Cold Weather Protection," Revision 10, provided guidance for the protection of system piping and components from damage caused by freezing temperatures.
Appendix A of Procedure 40OP-9ZZ17 contained historical data of record low temperatures that the licensee used as guidance to determine when freeze protection should be implemented.
The procedure directed the draining of various building heating, ventilation, and air conditioning cooling coils and air wash units.
It also and provided for the installation of temporary heat tracing, heat lamps, and insulation, as needed.
The licensee had effectively implemented this procedure.
The inspectors walked down portions of the refueling water, reactor makeup water, condensate storage and transfer tanks, and the ESP systems that are susceptible to freezing temperatures.
The inspectors observed that exposed piping containing liquids was provided with heat tracing and insulation. The heat tracing provided to these systems remained in service throughout the year and was provided with alarm indication to the control room.
Procedure 40OP-9ZZ17 also stated, in part, that the possibility of freeze damage occurring to the ESP and spray pond header risers had been evaluated by engineering
'1 I
-16-and was not needed.
The inspectors reviewed the ESP system piping and instrumentation drawings and determined that a check valve in the discharge header would prevent the spray header risers and nozzles from draining back to ESP lev'el. The ESP riser piping and nozzles were elevated above the bulk water volume of the ESP and would be exposed to freezing temperatures.
The inspectors reviewed the Updated Pinal Safety Analysis Report for the design basis consideration of potential freezing temperatures on plant equipment.
Section 2.4.7 stated, in part, that outdoor safety-related facilities were protected from subfreezing temperatures, pipes were installed underground, and the mass of water in the ESPs would not freeze because subfreezing temperatures have too short a duration.
The inspectors interviewed the ESP system design engineer and requested he provide the Procedure 40OP-9ZZ17 referenced engineering document that stated that freeze protection was not needed for the ESP riser piping and nozzles.
The licensee could not provide the referenced documentation.
The licensee provided a copy of Revision 0 of Procedure 40OP-9ZZ17 that included cross-department procedure review comments prior to the original version of the procedure being issued.
In the comments section, the Operations department had asked if freeze protection was needed for the spray header risers.
The answer stated that protection was not required.
The licensee initiated CRDR 1-8-0552 on December 17, 1998, to document this deficiency. Operations management issued an operations night order on December 18 to require monitoring of meteorological tower temperature.
If the temperature drops below 32'F, both ESP pumps shall be started and run until ambient temperature is above freezing.
Planned long-term corrective actions included completion of an engineering evaluation of the ice formation rate inside exposed ESP piping. Actions also required performing a probabilistic risk assessment for the increase in core damage frequency, based on the unavailability of the ultimate heat sink due to freezing of the exposed piping.
The inspectors reviewed the licensee's meteorological data covering the period of 1986 to present.
From this review, the inspectors determined that the maximum continuous time that ambient temperature remained below 32'F was 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br /> in 1990. Since 1990, the longest duration that temperature had remained below freezing was 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />.
Based on this information, the inspectors considered the licensee's corrective actions to be appropriate.
Conclusions The licensee effectively implemented the cold weather protection program to protect safety-related components and piping susceptible to freezing temperature ES Miscellaneous Engineering Issues (92904)
E8.1 Closed Ins ectorFollowu Item IFI 50-528529530/9805-05:
DesignModification Piping Material Information Incorrectly Translated Into Plant Change Work Orders (PCWO).
This item was opened pending the licensee's resolution of CRDR 9-8-0998. This CRDR was initiated to evaluate how incorrect material was translated from Design Master Work Order 800082 into the individual PCWO. The modification provided for the installation of an Annubar-style flow meter in the spent fuel pool cooling system discharge piping.
The design master work order specified piping material classification HCCA, which required the use of ASME Section III material. The implementing PCWO specified commercial-grade material.
The CRDR evaluation determined that the primary cause of the condition was inattention to details.
During the initial installation of the modification, it was discovered that the weld cap supplied by the vendor would not fitdue to distortion caused by welding on the 8-inch, schedule 20 stainless steel pipe. The situation was evaluated by an engineer, who supplied an alternate detail to be used in lieu of the vendor supplied cap. The alternate detail specified a standard 1-inch, 3000-Ib pressure rated, socket-weld stainless steel cap, piping class HCCA. The alternate detail was supplied to the planner, who made a pen and ink change to the PCWO. The pen and ink change specified APN 4355-328 instead of.APN 4355-321, which meets ASME Section III requirements.
The licensee's evaluation determined that a second party review of the original WO would have been appropriate considering the critical nature of the system.
The planner should have considered amending the WO and obtaining a formal technical review. The inspectors identified no problems with the licensee's evaluation.
IV. Plant Su ort R4 Staff Knowledge and Performance R4.1 Radiation Surve s and Boron Buildu Removal Units 1 and 2 ae Ins ection Sco e 71750 The inspectors identified minor boron buildup on both of the Unit 2 high pressure safety injection (HPSI) pump seal areas.
Discussions were held with radiation protection (RP)
personnel on how they monitor and remove boron buildup. The inspectors also identified a trace amount of what appeared to be boron on Valve 1SI-V840, a low pressure safety injection flush connection.
b.
Observations and Findin s On December 16, 1998, the inspectors identified minor boron buildup on the Unit 2 HPSI A and B pump seal areas.
The inspectors held discussions with RP personnel and determined that they performed weekly walkdowns of the auxiliary building areas to
A
-18-identify and monitor boron buildup on components.
RP personnel then evaluate the need to remove boron buildup and initiate work requests, as necessary.
In this particular case, the boron buildup on the HPSI pump seal areas had been identified and was being monitored by RP. The inspectors verified that active Work Requests 936099 and 952410 addressed the seal leaks.
The RP technician informed the inspectors that the boron buildup had been removed from the pumps on December 16. The inspectors returned to the pump areas and verified that the boron buildup had been removed.
On December 21, the inspectors identified what appeared to be a trace amount of boron on Unit 1 Valve Sl-V840. This was brought to the attention of RP personnel, who conducted a survey of the area.
No contamination was found.
Conclusions A good process for monitoring and removing boron buildup on safety-related components was implemented by the licensee.
V. Mana ement Meetin s X1 Exit Meeting Summary The inspectors presented the inspection results to members of the licensee's staff on January 6, 1999, after the conclusion of the inspection.
The licensee acknowledged the findings presented.
The inspectors asked the licensee whether any material examined during the inspection should be considered proprietary.
No proprietary information was identifie <I gl
f I
ATTACHMENT PARTIALLIST OF PERSONS CONTACTED Licensee M. Banks, Communication Representative, Owner Services S. Boardman, Section Leader, Maintenance Program D. Garnes, Unit 1 Department Leader, Operations D. Fan, Department Leader, Design Engineering Electrical/Instrument and Control R. Flood, Department Leader, System Engineering R. Fullmer, Director, Nuclear Assurance F. Gowers, Site Respresentative, El Paso Electric Co.
M. Heider, Section Leader, Engineering R. Henry, Site Representative, Salt River Project J. Hesser, Director, Engineering W. Ide, Vice President, Nuclear Engineering L. Johnson, Department Leader, Site Chemistry D. Kanitz, Engineer, Nuclear Regulatory Affairs P. Kirker, Unit 3 Department Leader, Operations A. Krainik, Department Leader, Nuclear Regulatory Affairs J. Levine, Senior Vice President, Nuclear D. Mauldin, Director, Maintenance D. Marks, Section Leader, Nuclear Regulatory Affairs G. Overbeck, Vice President, Nuclear Production M. Powell, Department Leader, Design Engineering T. Radke, Director, Outages M. Radspinner, Section Leader, Design Engineering J. Scott, Director, Chemistry D. Smith, Director, Operations S. Terrigrino, Department Leader, Strategic Communications P. Wiley, Unit 2 Department Leader, Operations M. Winsor, Department Leader, System Engineering
0'
t
-2-INSPECTION PROCEDURES USED 37551 61726 62707 71707 71750 92901 92903 Onsite Engineering Surveillance Observations Maintenance Observations Plant Operations Plant Support Activities Plant Operations Followup Engineering Followup ITEMS OPENED CLOSED AND DISCUSSED
~Oened 50-530/9809-01 50-528/9809-02 VIO NCV Operator failed to followprocedure for positioning of a SDC valve (Section 01.1)
Failure to properly monitor the PASS under the maintenance rule (Section M1.4)
50-528/9809-03 VIO Failure to properly monitor maintenance rule goals to ensure the PASS performs its intended function (Section M1.4)
Closed'0-530/98-01-00 50-528,529,530/9805-05 IFI LER TS 3.0.3 Entry Due To Safety Injection Flow Instruments Being Removed From Service (Section 08.1)
Design Modification Piping Material Information Incorrectly Translated Into PCWO (Section E8.1)
50-528/9809-02 NCV Failure to properly monitor the PASS under the maintenance rule (Section M1.4)
I f
-3-LIST OF ACRONYMS USED AO ASME auxiliary operator American Society of Mechanical Engineers BOP ESFAS balance of plant engineered safety features actuation system CFR CRDR CS DFWO EDG EDT EER ESP HPSI IBC LER MR MRRFF NRC PASS PCR PCWO PDR RCS RDT
",RP SDC TS WMPR WO Code of Federal Regulations condition report/disposition request containment spray deficiency work order emergency diesel generator equipment drain tank engineering evaluation request essential spray pond high pressure safety injection instrument and control licensee event report maintenance rule maintenance rule repetitive functional failure Nuclear Regulatory Commission postaccident sampling system plant change request plant change work orders Public Document Room reactor coolant system reactor coolant drain tank radiation protection shutdown cooling Technical Specifications west mechanical penetration room work order