IR 05000528/1998010
| ML17313A863 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 03/18/1999 |
| From: | NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML17313A862 | List: |
| References | |
| 50-528-98-10, 50-529-98-10, 50-530-98-10, NUDOCS 9903250220 | |
| Download: ML17313A863 (49) | |
Text
ENCLOSURE U.S. NUCLEAR REGULATORY COMMISSION
REGION IV
Docket Nos.:
License Nos.:
Report No.:
Licensee:
Facility:
Location:
Dates:
Inspector(s):
50-528 50-529 50-530 NPF-41 NPF-51 NPF-74 50-528/98-10 50-529/98-10 50-530/98-10 Arizona Public Service Company Palo Verde Nuclear Generating Station, Units 1, 2, and 3 5951 S. Wintersburg Road Tonopah, Arizona December 27, 1998, through February 6, 1999 J. H. Moorman, III, Senior Resident Inspector D. R. Carter, Resident Inspector D. E. Corporandy, Resident Inspector N. L. Salgado, Resident Inspector J. E. Whittemore, Senior Reactor Inspector Approved By:
P. H. Harrell, Chief, Project Branch D
. ATTACHMENT: Supplemental Information 9903250220 99'03'iB PDR ADQCK 05000528
I I
EXECUTIVE SUMMARY Palo Verde Nuclear Generating Station, Units 1, 2, and 3 NRC Inspection Report No. 50-528/98-10; 50-529/98-10; 50-530/98-10
~Oeratinna A failed gasket at a flanged connection caused a substantial leak in the Unit 3 instrument air header.
Operator actions to isolate the leak, combined with the proper functioning of the backup nitrogen supply, stabilized the instrument air header pressure to allow the unit to remain stable throughout this event. Actions taken by the operations staff to mitigate this event were effective (Section 02.1).
A sample of the Unit 3 emergency procedure appendices detailing the local manual operation of equipment by auxiliary operators during emergency conditions was reviewed.
It was determined that appropriate guidance for accomplishing the tasks had been provided (Section 03.1).
~
Control room operator response to the unexpected loss of the Unit 1 120-Vac Instrumentation Bus PND-D28 was good (Section 04.1).
Maintenance Knowledgeable technicians used approved procedures to perform routine maintenance activities in a safety conscious manner.
Good work and foreign material control practices were observed (Section M1.1).
Knowledgeable technicians used approved procedures to conduct surveillance activities in a safety-conscious manner (Section M1.2).
During routine tours, the observed material condition of components in all three units was good (Section M2.1).
Several deficiencies in the work control and planning process were identified by the licensee.
The combination of poor planning, engineering, technical reviews, verification inspection criteria, and verification specifications were identified.
In addition, a lack of periodic monitoring of modification activities and effective planner and worker training were noted.
These problem areas resulted in poor work planning in the plant design modification and welding areas.
Some personnel involved in the work planning process did not devote the required attention to detail to perform effective planning. Two noncited violations were identified. A violation of Criterion III occurred when a design error resulted in the installation of an improper component, which eventually failed. A violation of Criterion IX occurred when code-required nondestructive examination was not performed on modification welds prior to placing the system in service.
This resulted in a train of auxiliary feedwater later being declared inoperable until the examinations were performed and evaluated.
These noncited violations are being issued in accordance with Section VII.B.1 of the Enforcement Policy (Section M7.1).
r
-2-
~En ineerin
~
The licensee failed to maintain adequate design control measures when the cooling water line flexible joints for the emergency diesel generators were replaced.
The replacement joints were not preapproved for use.
The appropriate design measures were not taken to verify that the new joints could perform their intended safety function.
Failure of these joints would render the diesels inoperable.
The licensee promptly responded to address the issue and identified numerous other diesel expansion joint design discrepancies.
Upon completion of an evaluation, the licensee determined that
, the joints would perform satisfactorily and all emergency diesel generators remained operable.
This problem is a violation of Criterion III; however, this Severity Level IV violation is being treated as a noncited violation, consistent with Appendix C of the NRC Enforcement Policy (Section E3.1).
~'I f
i l
Re ort Details Summa of Plant Status Units 1, 2, and 3 operated at essentially 100 percent power for the duration of this inspection period.
Operational Status of Facilities and Equipment 02.1 Loss of Instrument Air IA Unit 3 a.
Ins ection Sco e 71707 On January 26, 1999, at approximately 5:52 a.m., the Unit 3 IA header experienced a substantial leak due to a failed gasket downstream of IA Dryer D. The valve configuration required to isolate the leak caused all three IA compressors to trip. The inspectors reviewed applicable unit togs, work orders (WO), procedures, and Condition Report/Disposition Request (CRDR) 3-9-0012.
b.
Observations and Findin s On January 26, control room operators dispatched an auxiliary operator (AO) to investigate the cause of IA alarms.
The AO identified a failed gasket at a flanged connection downstream of IA Dryer D after filteroutlet isolation Valve 3PIAN-VF52. The AO isolated the leak by closing the IAdryer isolation valves, which caused all three IA compressors to trip on high discharge pressure.
After the compressors had tripped, control room operators entered abnormal Procedure 40AO-9ZZ06, "Loss of Instrument Air," Revision 4. The operators used the procedure guidance appropriately.
Prior to this event, IA header pressure was normal at approximately 117 psig. During this event, when header pressure decreased to 85 psig, the nitrogen backup supply automatically initiated and stabilized IA pressure at 110 psig. The repair was classified as priority one, and maintenance personnel promptly removed and replaced the failed gasket with a 1/8-inch synthetic blended rubber gasket.
The IA system was realigned to its normal configuration at approximately 6:25 a.m.
Normal configuration means one compressor maintaining system pressure and the other two compressors in standby.
The unit remained stable throughout this event.
The licensee initiated CRDR 3-9-0012 to determine the failure mechanism of the gasket and the action required to prevent recurrence.
The CRDR would also determine transportability of the gasket failure to other IA flanges in all three units. The subject gasket had previously been replaced with a 1/8-inch silicone gasket, on March 6, 1997, via WO 786211.
This was done as part of a program to replace all 1/8-inch neoprene gaskets in the IA system.
The replacement effort was prompted by an age-related gasket failure on the Unit 2 IA system in December 1995.
The piping material classification drawings required that IA gaskets be 1/8-inch silicone.
The licensee initiated a deficiency work request to have engineering perform a formal evaluation of the acceptability of the current installation of the rubber gasket.
The
-2-evaluation of the acceptability of the current installation of the rubber gasket.
The licensee's preliminary evaluation determined that the gasket was acceptable for use until the formal evaluation was completed.
c.
Conclusions A failed gasket at a flanged connection caused a substantial leak in the Unit 3 instrument air header.
Operator actions to isolate the leak, combined with the proper functioning of the backup nitrogen supply, stabilized the instrument air header pressure
= to allow the unit to remain stable throughout this event. Actions taken by the operations staff to mitigate this event were effective.
Operations Procedures and Documentation 03.1 Local 0 eration Of E ui ment Durin Emer encies Unit3 Ins ection Sco e 71707 The inspectors reviewed and performed local walkdowns of emergency response procedure appendices requiring local-manual AO actions.
The inspectors evaluated access to equipment, equipment labeling, availability of emergency lighting, and access to special tools required to perform some specific procedure operations.
The inspectors evaluated the following appendices from Emergency Procedure 40EP-9EO10,
"Standard Appendices," Revision 12:
Appendix 10
"Charging Pump Alternate Suction to the Refueling Water Tank/
Restoration," Attachments 10-A, 10-B, and 10-C Appendix 11
"Charging Pump Alternate Suction to the Spent Fuel Pool/Restoration,"
Attachments 11-A and 11-B Appendix 18
"Local Atmosphere Dump Valve Operation," Attachment 18-C, "Local Operation of SGB-HV-178" Appendix 31 Appendix 32
"Local Monitoring of IA Pressure and Emergency Nitrogen Supply,"
Attachment 31-A, "Manual Operation of SGB-UV-130"
.
"Minimize Release to the Environment," Attachment 32-A Appendix 38
"Resetting AuxiliaryFeed Pump A," Attachment 38-A Appendix 39 Appendix 40 Appendix 41
"Local Operation of Auxiliary Feed Pump B," Attachment 39-A
"Local Operation of Auxiliary Feed Pump A," Attachment 40-A
"Local Operation of Auxiliary Feed Pump N," Attachment 41-A
-3-Appendix 42
"AligningAux Feedwater Pumps Suction to Reactor Makeup Water Tank," Attachment 42-A Observations and Findin s The emergency procedure appendices provided clear instructions for the local operation of plant equipment, as well as the location of some of the equipment.
Procedure 40EP-9EO10, Appendix 32, provided guidance on valves to be isolated to minimize the release of radioactivity to the environment.
The appendix described the general location of valves that would need to be closed, but did not provide the specific elevation of several valve locations.
In addition, Procedure 40EP-9EO10, Appendix 41, did not provide the location of the control power transfer switch for AuxiliaryFeedwater Pump N.
The inspectors questioned several AOs as to the specific location of these components.
The AOs questioned were able to describe the locations.
The inspectors determined that operator training provided the knowledge of these specific equipment locations complemented the guidance in the appendix.
Local valves and breakers were properly labeled with noun names and equipment identification numbers that were consistent with the emergency procedure.
All equipment that required operation was accessible to the operators.
Allareas, observed by the inspectors, were provided with essential/emergency lighting. The inspectors reviewed lighting diagrams to determine the location of emergency lighting in relation to the actual equipment that would have to be operated during an emergency.
The placement of this lighting appeared to be adequate for an operator to perform the intended equipment operation during a loss of power event.
Several of the appendices required special tools to perform designated tasks.
These procedures described the location of emergency equipment cabinets where required tools were located.
The inspectors inventoried the contents of Emergency Cabinet FPN-C02 for the tools required to manually charge the closing springs for Breaker PBB-SO4S for AuxiliaryFeedwater Pump B. The cabinet was organized and all tools were labeled appropriately.
Conclusions A sample of the Unit 3 emergency procedure appendices detailing the local manual operation of equipment by AOs during emergency conditions was reviewed.
It was determined that adequate guidance for accomplishing the tasks had been provided.
Operator Knowledge and Performance 04.1 Control Room 0 erator Res onse to Loss of Vital 120-Vac Instrument Bus PND-D28 Unit 1 Ins ection Sco e 71707 On January 12, 1999. the inspectors responded to the Unit 1 control room after the output breaker to Inverter PND-N41 tripped open, causing a loss of power to vital
i w
120-Vac Instrument Bus PND-D28. The inspectors observed the crew's response to the de-energized instrument bus, as well as subsequent actions to re-energize the instrument bus and troubleshoot the cause.
b.
Observations and Findin s At 12:48 p.m. on January 12, 1999, the Inverter PND-N41 output breaker unexpectedly opened, which resulted in a loss of power to vital Instrument Bus PND-D28. Reactor Trip Breakers B and D opened due to a toss of power to reactor protection system logic Matrices AD, BD, and CD, which generated expected engineering safeguards actuation signal half-leg trips. Control room operators entered Procedure 40AO-9ZZ13, "Loss of Class Instrument Power," Revision 2. The inspector verified that the licensee appropriately entered into numerous Technical Specification (TS) action statements due to the loss of the Core Protection Calculator D, the core operating limitsupervisory system, and the plant computer.
Entering TS 3.8.4, "DC Sources - Operating," placed the plant in a 2-hour limiting condition of operation for restoration of power to the bus.
The instrument bus was re-energized from its associated voltage regulator at 2:41 p.m.,
within the 2-hour limiting condition of operation requirement.
The licensee conducted troubleshooting activities and determined that the inverter alarm logic card had failed.
The alarm card was replaced, and the inverter was returned to operation.
Prior to the loss of Instrument Bus D, Channel A of the plant protection system was declared inoperable because all parameters of Channel A were placed in bypass to perform scheduled surveillance Procedure 36ST-9SB02, "PPS Bistable Trip Units Functional Test," Revision 18. At 1:59 p.m., the licensee backed out of surveillance Procedure 36ST-9SB02, removed Channel A from bypass, and then placed all parameters of Channel D into bypass in accordance with Procedure 40AO-9ZZ13.
The licensee initiated WO 869584 as priority one so the troubleshooting action plan could be performed.
A sensitive issues briefing was conducted with cognizant work groups, operations personnel, and'the site shift manager.
The briefing reviewed the troubleshooting action plan, discussed possible contingency actions, and was conducted with a questioning attitude by participants.
The control room supervisor (CRS) conducted frequent briefings to update the control room operators on significant issues.
The shift manager maintained good oversight of overall plant activities and the restoration of the instrument bus, while the CRS addressed the loss of the instrument bus on the plant and the performance of required surveillance procedures due to the loss of the instrument bus.
The licensee initiated CRDR 1-9-0006 to document the cause of the unexpected opening of the inverter output breaker and address transportability to other equipment.
Conclusions Control room operator response to the unexpected loss of the Unit 1 120-Vac Instrumentation Bus PND-D28 was goo Quality A'ssurance ln Operations 07.1 CRDR Review Ins ection Sco e 71707 During this inspection period, the inspectors reviewed CRDRs 3-6-0114 and 3-8-0237 to determine if corrective actions had been completed.
Observations and Findin s CRDR 3-6-0114 evaluated the cause of the Unit 3 emergency diesel generator {EDG)
day tank being inadvertently drained below the minimum TS level ~ The inadvertent drain occurred during the performance of Procedure 73ST-9DF01A, "Diesel Fuel Oil Transfer Pump Inservice Test." Procedural steps required lowering tank level by opening the drain valve on the day tank and subsequently closing the drain valve when the pump was verified to start at the local panel.
During performance of the test, the light bulb in the local "red" running light was missing. With no light to indicate that the pump had started, the drain valve was not closed in a timely manner.
The drain rate exceeded the makeup capacity of the transfer pump; consequently, level in the day tank decreased.
The control room received the day tank low level alarm, and the AO was contacted to close the drain valve. The inspectors reviewed the licensee's corrective actions and determined them to be comprehensive and appropriate.
No similar events have been identified since completion of the corrective actions.
CRDR 3-8-0237 described a condition in which Control Element Assembly {CEA) 14 was unintentionally moved three steps during the performance of a portion of Procedure 36MT-9SF15, "CEA Coil Traces at Power Operation," Revision 6. The unintentional movement of CEA 14 was attributed to the failure of an operator to transfer CEA 14 to its hold bus prior to pulsing it. This occurred because of a miscommunication between:
control room personnel; the CRS, a reactor operator, and personnel in the control element drive mechanism control system room; and an AO and an instrument and control technician.
The inspectors verified that the corrective actions were comprehensive and appropriate.
With the exception of a commitment to include a discussion of the event in an industry events training session, all corrective actions had been completed.
The due date for this commitment was March 30, 1999. The corrective actions included coaching of the individuals involved in the event, briefing of the operations crew, and the addition of a procedure step requiring additional verification that CEAs have been transferred to or from the hold bus as appropriate.
No similar events have been identified since completion of the corrective actions.
Conclusions Corrective actions for two CRDR reviewed by the inspectors were comprehensive and appropriat '1
l
-6-
Miscellaneous Operations Issues (92901)
08.1 Ino erableHi hPressure Safe Inection HPSI S stemlssues Units1 2 and3 On December 21, 1998, a Notice of Violation and Proposed Imposition of Civil Penalty was issued based on apparent violations (APV) discussed at a predecisional enforcement conference held in the NRC Region IV office on September 14, 1998 (reference NRC Inspection Report 50-528;50-529;50-530/98-14).
In the Notice of Violation, the apparent violations discussed in Sections 08.2 through 08.5 were dispositioned.
The following items are being opened in this inspection report, for administrative purposes, to provide tracking numbers for those violations:
~
VIO 50-528;50-529;50-530/981 4-06: Five Examples of Failure To Meet TS 3.5.2
~
VIO 50-528;50-529;50-530/9814-07:
Two Examples of HPSI Pump Discharge Check Valve Conditions Adverse To Quality
~
VIO 50-528;50-529;50-530/9814-08:
Two Examples of Inadequate HPSI Discharge Check Valve Procedures 08.2 Closed APV 50-528 50-529 50-530/9814-01:
Three Examples of an Apparent Violation of TS 3.0.3 This item is being administratively closed and being tracked by the violations defined in Section 08.1 of this inspection report.
08.3 Closed APV 50-528 50-529'50-530/9814-02:
Two Examples of An Apparent Violation of TS 6.8.1 As described in a letter from the NRC to the licensee, dated December 21, 1998, this APV was not cited because it was considered to be another example of the violation cited in NRC Inspection Report 50-528;50-529;50-530/98-11.
08.4 Closed APV 50-528 50-529 50-530/9814-03:
Five Examples of APV of 10 CFR Part 50, Appendix B, Criterion XVI This item is being administratively closed and being tracked by the violations defined in Section 08.1 of this inspection report.
08.5 Closed APV 50-528 50-529 50-530/9814-05:
Two Examples of APV of TS 3.5.2 This item is being administratively closed and being tracked by the violations defined in Section 08.1 of this inspection repor.6 Closed LicensedEventRe ort LER 50-528/98-002:
Reactor TriponLowSteam Generator Level due to Insufficient Feedwater Flow Followed by an AuxiliaryFeedwater Actuation Signal On February 22, 1998, Unit 1 experienced a reactor trip due to the failure of the dynamic compensator card in the level compensator circuitry for the Steam Generator 2 feedwater control system.
Licensee response to the event was good, as detailed in NRC Inspection Report 50-528;50-529;50-530/98-02.
The licensee conducted a reactor trip investigation and performed a root cause determination on the failed compensator card. The card was replaced and retested.
The licensee evaluated the transportability of the failure to other systems in all units that use this type of compensator card. The root cause could not determine a specific failure mechanism, and a review of prior card failures showed only three repairs of this type of card since 1990.
Licensee long-term corrective actions were to replace the feedwater control systems in all units with a new fault-tolerant digital electronic system.
Unit 1 is scheduled for this modification during the next refueling outage.
II. Maintenance M1 Conduct of Maintenance M1.1 General Comments on Maintenance Activities Units 1 2 and 3 a.
Ins ection Sco e 62707 The inspectors observed all or portions of the following work activities:
954356
"Install High Pressure Safety Injection Pump Bearing Modification As Directed By DMWO 870115" (Unit 3)
867690
"Troubleshoot, Rework and/or Replace MSFIS Logic 30 Volt Power Supply" (Unit 3)
867637
"2MCHBPO1 - Perform Plunger Packing and/or Plunger Replacement" (Unit 2)
861277
"Replace Two Swagelok Fittings in Essential Cooling Water System Train B" (Unit 1)
b.
Observations and Findin s The inspectors found the work performed under these activities to be properly performed.
Allwork observed was performed with the work package present and in
~
active use.
Work and foreign material exclusion practices observed were good.
Technicians were experienced and knowledgeable of their assigned task c.
Conclusions Knowledgeable technicians used approved procedures to perforrri routine maintenance activities in a safety-conscious manner.
Good work and foreign material control practices were observed.
M1.2 General Comments on Surveillance Activities Units 2 and 3 a.
Ins ection Sco e 61726 The inspectors observed all or portions of the following surveillance activities:
"Diesel Fuel Oil Tank Condensate Removal," Revision 1 (Unit 2)
"Diesel Fuel Oil Transfer Pump - Inservice Test," Revision 6 (Unit 2)
"HPSI Pumps Miniflow - Inservice Test," Revision 10 (Unit 3)
b.
Observations and Findin s The inspectors found that knowledgeable personnel performed these surveillances satisfactorily, as specified by applicable procedures.
'onclusions Knowledgeable technicians used approved procedures to conduct surveillance activities in a safety-conscious manner.
M2 Maintenance and Material Condition of Facilities and Equipment M2.1 Review of Material Condition Durin Plant Tours Units 1 2 and 3 a.
Ins ection Sco e 62707 h
During this inspection period, routine tours of all units were conducted to evaluate plant material condition.
b.
Observations and Findin s Inspectors observation of plant material condition during this inspection period identified no major observable material condition deficiencies.
Minor deficiencies, brought to the attention of the licensee, were documented with work requests.
c.
Conclusions During routine tours, the observed material condition of components in all three units was goo 'I
l I
(
-9-M7 Quality Assurance ln Maintenance Activities M7.1 Review of Work Control and Plannin Activities Units 1 2 and 3 Ins ection Sco e 93902 The inspectors reviewed the planned programmatic changes to the work control and planning processes.
To ensure that the current processes were adequate, the inspectors reviewed a sample of WOs and other work control documents where corrective action had been taken to address planning and control issues.
Procedures related to the control of work and welding were also reviewed.
b.
Observations and Findin s
~
Dela ed Corrective Action On June 11, 1998, the licensee became aware that the processing of CRDRs for work control and planning issues were being delayed because of supervisory handling of corrective action documents.
Procedure 90DP-OIP10, "Condition Reporting,"
Revision 4, recommended that supervisory processing be completed within 3 working days.
The licensee initiated CRDR 9-8-0971, which identified six cases where supervisory personnel did not review and process CRDRs within the expected time of 3 days after receiving a CRDR. The affected CRDRs were 9-8-0919, 9-8-0924, 9-8-0937, 9-8-0938, 9-8-0939, and 9-8-0940.
The largest delay in processing these CRDRs was related to CRDR 9-8-0937, which was discovered, initiated, and given to the supervisor on March 21, 1998, and signed by the supervisor on June 12, 1998.
The inspectors reviewed the completed work documents and other CRDRs associated with the untimely CRDRs. The six associated tasks consisted of four that were safety
~
related and two that were not safety related.
The inspectors noted that the CRDRs issued for all related WOs reported administrative deficiencies that resulted from a poor planning effort. The corrective actions taken addressed the identified WO deficiencies and performed the necessary reviews to identify any generic issues that had resulted in the CRDRs being issued.
The inspectors reviewed the final condition of the tasks identified on the CRDRs to ensure that there were no safety concerns or operability issues as a result of the work performed.
An example of the licensee's treatment of these issues was the handling of CRDR 9-8-0938, which identified concerns with WO 767819.
This WO was issued for Unit 1 to replace a leaking spring check valve installed on a bonnet bleed line for motor-operated Valve 1JSIBUV0656 (Shutdown Cooling Loop 2 Containment Isolation).
Amendment D to the WO was issued to replace the check valve a second time when the first replacement valve did not meet the local leak rate testing administrative criteria.
During installation of the second check valve, the body-to-bonnet threads became damaged (galled) and a decision was made to leave the damaged check valve in place.
The CRDR reported that the WO did not justify leaving the valve in place, WO steps were deleted by pen and ink change without the proper reference, a corrective
-10-maintenance WO was issued instead of a deficiency WO to repair the valve, and the vendor or technical drawings were not referenced to provide acceptance criteria for thread restoration.
The inspectors verified through a review of documents, that engineering had reviewed and evaluated, as satisfactory, the valve leakage identified during testing and the adequacy of'the damaged check valve threads to function as required.
Additionally, the inspectors became aware of CRDRs 9-8-0515 and 9-8-1179, which had been issued by management to address concerns about the licensee's welding and work control planning program.
The CRDRs contained action items designed to:
~
Revise the weld work planning process from the engineering design process to field implementation, Define the allowable scope of pen and ink changes to WOs, Develop an Engineering/Maintenance Welding Installation/Inspection Handbook, Provide training to planners regarding the proper identification of WO references for information and implementation, Strengthen the WO planning technical review process, As an interim action, perform a re-review of all Unit 3 refueling outage safety-related work packages prior to being released to perform work in the field, and Rewrite Specifications 13-PN-204 and 13-PN-205 to improve understandability, usability, and job compatibility.
The inspectors determined that the specific WO planning issues identified in the six untimely CRDRs had been adequately addressed and there were no current safety issues identified with these tasks.
In addition, the programmatic deficiencies identified were to be addressed in CRDRs 9-8-0515 and 9-8-1179.
Im ro er Material Sta ed for Installation During field implementation of WOs 804325 and 804327 in Unit 2, it was discovered that piping elbows of the wrong material were about to be installed in the nonsafety-related main turbine moisture separator reheater drain system.
A planner in the field identified that chrome molybdenum, instead of the required carbon steel elbows, were issued for installation. CRDR 9-8-0947 was initiated to address this issue.
Licensee personnel attributed the cause of the near miss to be personnel error by the planner, material coordinator, or both. Transportability and technical reviews of work packages that had been planned or supplied by these two individuals were conducted to identify any other improper material issues.
The review identified two other problems,
I
-11-and CRDRs 9-8-0613 and 9-8-0919 were initiated to address the problems.
The inspectors verified through a review of documents that the correct material had been installed.
No safety issues were identified during review of this item.
Wron Size Gasket Installed A Unit 2 modification was installed under WO 717994 to prevent pressure locking of motor-operated Valve 2JSIDUV0654 (Loop 2 Shutdown Cooling Containment Isolation Valve). After the installation, one of the 3/4-inch flanged joints on each side of a check valve, installed as part of the modification, failed and leaked excessively.
The leak was repaired under WO 812295, after a determination was made that the joints were
.
assembled with the incorrect spiral-wound gaskets.
The correct gaskets were then installed.
CRDR 9-8-0966 was initiated and the licensee's review identified the cause of the problem as personnel error on the part of the design engineer.
During the design of the modification, the designer specified dimensions for a gasket that did not exist. When the design requirements were translated into an inventory part number, the wrong gasket was selected and issued for installation. The inspectors determined that the final modification installation for this particular application was appropriate and there were no safety concerns.
However, under the design master work order (DMWO), the licensee had implemented five additional WOs to install the modification in five similar applications in the three units. As a result, the licensee conducted a transportability review and evaluated the completed WOs. The other application in Unit 2 was determined to have the correct gasket.
The review revealed that the valves in Unit 3 contained the incorrect gaskets, which were replaced with the correct gaskets during the previous outage.
The review also revealed that one of the two modifications in Unit 1 contained the incorrect gaskets, which were scheduled for replacement during the upcoming outage.
Thus, the licensee's organization had implemented a safety-related modification with improper material because of personnel error in the implementation of the design process.
The failure to establish adequate measures for the selection for review and suitability of application of materials, parts, equipment, and processes that are essential to the safety-related functions of the structures, systems, and components is a violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control. This nonrepetitive, licensee-identified and corrected violation of Appendix B to 10 CFR Part 50 is being treated as a noncited violation, consistent with Section VII.B.1 of the NRC Enforcement Policy (50-528,50-529,50-530/9810-01).
Deficient WO 717992 WO 717992 was approved and issued to implement DMWO 714670 in Unit 1 to install a spring-loaded check valve on the bonnet of motor-operated Valve 1JSIDUV654 (Shutdown Cooling Isolation Loop 2). The purpose of the DMWO was to install a modification to relieve pressure locking of motor-operated valves.
Licensee personnel identified the following deficiencies in the approved implerrienting WO:
-12-
~
There were several deficiencies that should have been identified during the technical review process,
~
There were excessive pen and ink changes to the document,
~
Some of the technical reviewers did not have sufficient experience in the mech'anicaI/welding planning area to perform the required technical review,
~
The wrong size bolting and required fastener torque was specified for the flanged connection, and
~
A WO step required drilling a hole in 16-inch pipe to accept the loss of a half coupling that was rated for 3000 psi instead the required 6000 psi..
Licensee personnel dispositioned the hardware issues through corrective actions developed in CRDRs 9-8-0425 and 9-8-0940. The inspectors verified through a review of documents that the correct flange fasteners were installed, required torque was applied to the fasteners, and the proper size and rated coupling half was installed. The programmatic issues related to pen and ink changes and inadequate technical review had been addressed or were being addressed in CRDRs 9-8-0367, 9-8-0515, and 9-8-1025. CRDR 9-8-0515 remained open to address programmatic issues.
The inspectors did not identify any safety concerns with the modification, as installed.
Deficient WOs 767831 and 767833 WOs 767831 and 767833 implemented a design change to Unit 1 AuxiliaryFeedwater Valves 1JAFBUV0035 (AuxiliaryFeedwater Pump B to Steam Generator 2 isolation)
and 1JAFBUV0036 (auxiliary feedwater to steam generator 1 downstream valve). The purpose of the modification was to prevent pressure locking of the safety-related, motor-operated valves.
The following problems were identified with these work documents:
~
Check valves that were to be installed in the motor-operated valve bleed off lines were described as relief valves in the work documents,
~
The WO was not clear about the required postmodification pressure testing and responsible test personnel, and
~
These and other deficiencies were missed in the required technical review.
Through a review of documents, the inspectors determined that the correct components (check valves) had been installed and the required pressure testing had been performed by qualified personnel.
However, during a transportability review to the other units, the licensee could not locate records to verify that some of the required nondestructive examination had been performed on the identical Unit 3 modification.
The licensee initiated CRDR 3-8-0116 when it was determined that the ASME-required preservice inspection dye penetrant tests had not been performed and documented for five welds made during the modification in Unit 3. The immediate action was to declare
-13-AuxiliaryFeedwater System Train B inoperable and perform the required testing on the five welds. A root cause analysis was performed that identified the cause of failure to be inattention to detail, with contributing causes identified as complacency, a large repetitive work load; and weld data sheets that were difficultto fillout and review. The licensee prescribed programmatic corrective actions in CRDRs 9-8-0515 and 9-8-1179 that were still in process.
The inspectors did not identify any safety concerns regarding the modification in all the units, as currently installed and tested.
However, the licensee had designed, implemented, and placed in service, a Unit 3 safety-related modification and failed to perform the ASME Section XI code required nondestructive examination of pressure boundary welds.
The failure to establish measures to assure that special processes, including welding, heat treating, and nondestructive testing, are controlled and accomplished by qualified personnel in accordance with applicable codes, standards, specifications, criteria, and other special requirements is a violation of 10 CFR Part 50, Appendix B, Criterion IX, Control of Special Processes.
This nonrepetitive, licensee-identified and corrected violation of Appendix B to 10 CFR Part 50 is being treated as a noncited violation, consistent with Section VII.B.1 of the NRC Enforcement Policy (50-528;50-529;50-530/9810-02).
Deficient WOs 741775 and 768391 The licensee's organization identified and corrected numerous deficiencies in these WOs following technical review, approval, and work implementation.
The issues were mainly related to welding, and the deficiencies were corrected when identified. Licensee personnel decided that the corrective actions related to the weak technical review process were already being addressed by CRDR 9-8-0515. The inspectors verified through a review of documentation that the final task performance and results were appropriate.
No safety concerns were identified for either task.
Poor Work Performance Durin Im lementation of WO 737059 This WO implemented a modification to install a nonsafety-related steam generator overboard discharge system that would provide the ability to discharge steam generator blowdown liquid directly to the retention basin.
Following an assigned walkdown by
, licensee personnel, management became aware that some of the newly installed piping hangars and supports had not been properly inspected by contractor personnel when installing the new system.
Following this discovery, CRDR 9-8-1179 was initiated. Aside from the personnel error to properly verify critical attributes associated with the supports and hangars, licensee personnel also identified programmatic elements in the nonquality-related verification process that required improvement.
These elements included poor verification inspection criteria, poor verification specifications, lack of periodic monitoring of modification activities, and lack of effective planner and worker training.
CRDR 9-8-1179 remained open pending identification and completion of all the programmatic corrective action needed to improve the nonquality-related verification inspection process.
No safety concerns were identified during review of this ite Additional Deficiencies in WO 767819 CRDRs 9-8-0424 and 9-8-0939 identified the following additional concerns related to the adequacy of the WO 767819:
Several deficiencies in the WO were identified and corrected by rewriting the WO,
'ngineering and planning technical reviews were not performed on the rewritten WO, A conflict on the design validation testing between DMWO 746112 and modification implementing WO 767819 was noted, and
~
WO 767819, Step 3.33, required l~-inch flange bolting material; DMWO 746112 required the installation of 5/8-inch bolting material.
An engineering and a planner re-reviewed WO 767819 and verified that all corrections met the guidelines found in Procedures 30DP-OAPO1, "Maintenance Instruction Writer'
Guide," Revision 22, and 30DP-9WPO2, 'Work Document Development and Control,"
Revision 25.
To resolve the testing requirements conflict, the planning department reviewed the test requirements and removed a requirement to perform a surveillance leak test. The normal preservice pressure test and code-required examinations were conducted, as appropriate.
The licensee verified that the proper 5/8-inch bolting material was used and torqued to the proper value.
Licensee personnel reviewed 41 packages prepared by the same planner to determine if there were generic issues related to bolting material or required torque values and did not identify further problems.
Through a review of documentation, the inspectors determined that the licensee had installed the modification in accordance with the appropriate design and safety requirements.
No current or additional safety issues were identified during review of this WO.
Personnel Performance In addition to the process weaknesses that have been identified, the licensee identified occurrences where personnel involved in the planning and work processes had not devoted sufficient attention to detail. Licensee management and supervision have provided counseling to applicable personnel to improve performance in this area.
Conclusions The licensee failed to maintain design control measures when the cooling water lines for the emergency diesel generators were replaced.
The joints were not like-for-like
f
-15-replacements, and the appropriate design measures were not taken to verify that the new joints could perform their intended safety function.'ailure of these joints would render the diesels inoperable.
This NRC-identified problem is a violation of Criterion III, with no response required.
The licensee promptly responded to address the issue and identified numerous other diesel expansion joint design discrepancies.
These discrepancies were indicative of unacceptable design control with regard to the diesel expansion joints. Upon completion of an evaluation, the licensee determined that the joints would perform satisfactorily.
MS Miscellaneous Maintenance issues (92902)
M8.1 Closed Violation VIO 50-528/9803-03:
EDG Internal Inspections Performed Without Proper Authorization.
This violation involved engineering personnel who performed boroscope inspection on the Unit 1 EDG A cylinder liners without obtaining proper authorization.
The inspectors verified the corrective actions described in the licensee's response letter, dated June 25, 1998, to be reasonable and complete.
No similar problems were identified.
III. En ineerin E3 Engineering Procedures and Documentation E3.1 Documentation Problems and Installation Inconsistencies with Ex ansion Joints on the EDGs Units 1 2 and 3 a.
Ins ection Sco e 37551 71707 On December 7, 1998, while touring the Unit 2 EDG rooms, the inspectors identified gaps on the tie rods for the 6-inch nominal diameter expansion joints on the essential spray pond (ESP) coolant lines from the EDG intercoolers.
The inspectors questioned the purpose and validity of the gaps.
The inspectors'eview included interviewing responsible licensee personnel and reviewing documentation including drawings, calculations, and the 10 CFR 50.59 screening.
b.
Observations and Findin s Following the inspectors'dentification of the EDG expansion joint tie rod gaps, the
'icensee checked the gaps on the ESP coolant lines from the EDG intercoolers at all six locations (one expansion joint on each diesel - an EDG A and B on each of the three units). The gaps were inconsistent among EDG expansion joints, varying from less than 1/8-inch up to approximately 3/4-inch.
Following a review of the expansion joint drawings, which indicated that a gap of 1 inch was allowed, discussions with Cooper Bessemer (the EDG manufacturer), and based on past successful runs of the EDGs, licensee engineers concluded that the gaps we'e within the design allowables, hence an operability evaluation was not necessar l p
e-16-However, the inspectors identified the following concerns:
~
The expansion joint drawings, which specified the gaps on the tie rods, represented the maximum allowed axial movement in order to'preserve the integrity of the expansion joint component.
It was unclear by the drawing if the maximum axial movement was allowed concurrently with the allowable lateral movement (i.e., the lateral movement required in the EDG application could reduce the allowable axial gap.) Furthermore, satisfaction of expansion joint vendor allowables did not ensure that allowable loads at the intercooler nozzle would be satisfied.
Expansion joint blow apart forces needed to be considered if the tie rods were not snug.
These are the unbalanced forces resulting from the differential pressure between. the water pressure inside the expansion joint and ambient air pressure outside of the expansion joint applied over the effective area of the expansion joint.
~
The licensee identified that the expansion joints observed by the inspectors were replacement expansion joints. The currently installed expansion joints were of a different type with different performance parameters than the original expansion joints (e.g., axial stiffness of the replacement expansion joints was approximately 40 percent higher than that of the original expansion joints, 2457 Ibfin versus 1750 Ibfin). The calculation of record had been performed using the performance parameters of the original expansion joint.
The inspectors considered that the discrepancies between the as-calculated and as-built ESP coolant lines would affect the fatigue performance of the system, a long-term performarice concern.
However, the inspectors did not consider the discrepancies to
-
represent an immediate operability concern because prior tests on all EDGs provided assurance of operability. Furthermore, the expansion joint loading was secondary in nature.
The basic characteristic of a secondary load is that it is self limiting. Local yielding can satisfy the conditions that cause the load to occur, and failure from a low number of cycles would not be expected.
After a detailed review of System Design Calculation 13-MC-SP-502, Revision 11, licensee engineers identified that system allowable loads would be exceeded with the 1-inch gap allowed on the drawing for the expansion joints. However, since the as-built gaps observed for the six expansion joints were all significantly less than 1-inch, the licensee again considered each of the EDGs to be operable.
The inspectors considered the 1-inch tie rod gap shown on the expansion joint drawings to constitute inaccurate information. The range of values chosen for controlling the parameters related to acceptable expansion joint loads, as determined by the system design calculations, was not correct.
The inspectors reviewed Engineering Evaluation Number 02157, Revision 0, which provided the evaluation for using the replacement expansion joints.
This evalation did not model stiffness characteristics and other pertinent parameters of the replacement expansion joints and did not consider the effects of expansion joint loading on the adjacent intercooler nozzle The specific concern identified with this issue was that a critical safety component was installed in a risk-significant system without adequate prior review to determine if the component could perform its intended safety function. This represented a failure of the design control program to ensure that the original design basis was maintained when installing previously unapproved components.
In addition, instructions provided for installation of the component were not adequate.
The failure to maintain design control measures commensurate with those applied to the original design is a violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control." This Severity Level IVviolation is being treated as a noncited violation, consistent with Appendix C of the NRC Enforcement Policy. This violation is in the licensee's corrective action program as CRDR 9-8-1856 (50-528;50-529;50-530/9810-03).
In parallel with the investigation on the EDG expansion joint tie rod gaps, the system engineer and design engineer promptly researched information on the other 6-inch diameter EDG expansion joints.
In addition to the expansion joints on the ESP coolant lines from the intercoolers, each EDG also has an expansion joint on the following lines:
engine-driven jacket water pump discharge (with tie rods), jacket water supply to the engine (no tie rods), lube oil supply to the engine (no tie rods), and engine-driven lube oil pump discharge (with tie rods). This means that there are 24 locations with 6-inch diameter EDG expansion joints, besides the six ESP coolant EDG expansion joints.
The following additional EDG expansion joint issues were identified by the license:
Currently, 23 of these 24 EDG 6-inch diameter expansion joints are the originally installed 4 convolution expansion joints manufactured by Tube Turns. One expansion joint, located on the engine-driven lube oil pump discharge to Unit 3 EDG B, is a 20 convolution expansion joint manufactured by Temp Flex. A suitable calculation existed for the lines with the 4 convolution expansion joints, but no calculation exists for the 20 convolution expansion joint.
The expansion joint currently being used for the ESP coolant (an 18-convolution, 6-inch diameter, Senior Flexonics, expansion joint) from the intercoolers was also approved as a replacement part for the aforementioned 6-inch diameter EDG expansion joints. The expansion joint drawing does not identify at which locations tie rods should be removed, nor does a suitable calculation exist to demonstrate the adequacy of the Senior Flexonics expansion joints as a replacement.
~
The SIMS (station information management system) database for the EDG expansion joints did not reference the appropriate drawings for the joints.
By the close of the inspection period, the licensee had accomplished the following corrective actions:
~
After a preliminary evaluation, Engineering revised the drawing for the expansion joints on the ESP coolant from the intercoolers to correct the tie rod gap dimension to "0."
-18-Maintenance personnel snugged the tie rod gaps at each of the six expansion joints to comply with the drawings.
By the close of the inspection period, the licensee had completed plans to accomplish the following corrective actions:
Procurement Engineering willcommunicate the drawing errors to the expansion joint manufacturer, Senior Flexonics.
Procurement Engineering will investigate as to why the instruction to snug the tie rods was not communicated on the Senior Flexonics drawing.
Engineering willadd appropriate instructions as to which applications require removal of the tie rods on the Senior Flexonics expansion joints.
The SIMS data base willbe updated to reference the appropriate expansion joint drawings for the EDG expansion joints.
Engineering willcomplete calculations to demonstrate the adequacy of the as-found gaps on the expansion joints on the ESP coolant lines from the intercoolers.
Engineering willcomplete calculations reflecting the current as-built condition (no gap at the Senior Flexonics expansion joint tie rods) on the ESP coolant lines from the intercoolers.
Engineering will,complete calculations/evaluations to demonstrate the adequacy of the as-built configuration of the engine-driven lube oil discharge line on Unit 3 EDG B.
Engineering willcomplete calculations/evaluations to demonstrate the adequacy of the Senior Flexonics expansion joints as possible future replacements for the other expansion joints.
c.
Conclusions The licensee failed to maintain adequate design control measures when the cooling water line flexible joints for the EDGs were replaced.
The replacement joints were not pre-approved for use.
The appropriate design measures were not taken to verify that the new joints could perform their intended safety function. Failure of these joints would render the diesels inoperable.
The licensee promptly responded to address the is'sue and identified numerous other diesel expansion joint design discrepancies.
Upon completion of an evaluation, the licensee determined that the joints would perform satisfactorily and all EDGs remained operable.
This problem is a violation of Criterion III; however, this Severity Level IVviolation is being treated as a noncited violation, consistent with Appendix C of the NRC Enforcement Polic 'T
-19-EB Miscellaneous Engineering Issues (92903)
E8.1 Closed VIO 50-529/9715-03:
Failure to Follow Procedures
, This violation involved the failure of engineering personnel to adhere to Procedure 70TI-9SP03, "Chemical Passivation of Spray Pond Piping," Revision 0. This failure potentially compromised the safety of personnel performing the work in the essential spray pond. The inspectors verified the corrective actions described in the licensee's response letter, dated October 10, 1997, to be reasonable and complete.
No similar problems were identified.
IV. Plant Su ort P2 Status of Emergency Preparedness Facilities, Equipment, and Resources P2.1 Offsite Sirens Fail Durin Testin a.
Ins ection Sco e 71750 During this inspection period, the inspectors reviewed the circumstances surrounding three separate failures of the offsite siren system to function during routine testing. The inspectors reviewed the licensee emergency plan, applicable contingency procedures, and associated CRDRs.
b.
Observations and Findin s The Unit 1 control room'was notified by the Maricopa County Sheriff's Office on three occasions of failure, of the offsite siren system.
See NRC Event 35166, 35174, and 35211 for additional information. For each of the failures:
The discovery of the condition was identified during routine testing of the offsite siren system by Maricopa County Sheriffs Office personnel.
Emergency plan personnel notified the licensee's communication personnel, who promptly corrected the problem.
Emergency plan personnel implemented contingency measures should the need arise to alert the public in the emergency planning zone of an emergency condition.
~
Notification to the NRC was made within the 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> requirement.
All three failures occurred at the licensee's microwave center, located in Phoenix.
This center translates a signal to a transmitter, which provides a signal to the sirens.
The licensee entered the failures into their corrective action program. After the latest failure
-20-(January 1, 1999), the licensee initiated significant CRDR 1-9-0001, which recommended a root cause failure analysis be performed and the preventive maintenance frequency be reviewed.
Prior to the recent failures, the offsite siren system performance had been very reliable.
c.,
Conclusions
After three failures of the offsite siren system, the licensee was aggressively looking at the system to ensure continued reliable performance.
V. Mana ement Meetin s X1 Exit Meeting Summary The inspectors presented the inspection results to members of the licensee's staff at the conclusion of the inspection on February 11, 1999. The licensee acknowledged the findings presented.
The inspectors asked the licensee whether any material examined during the inspection should be considered proprietary.
No proprietary information was identifie II l
e
ATTACHMENT PARTIALLIST OF PERSONS CONTACTED Licensee M. Banks, Communications Representative, Owner Services/Communication S. Burns, Department Leader, Maintenance Engineering B. Drost, E80 member, Salt River Project D. Fan, Department Leader, Design Engineering Electrical and 18C R. Fullmer, Director, Nuclear Assurance F. Gowers, Site Representative, Ef Paso Electric R. Hazelwood, Engineer, Nuclear Regulatory Affairs M. Heider, Section Leader, Procurement Engineering A. Krainik, Department Leader, Nuclear Regulatory Affairs D. Mauldin, Director, Maintenance M. Melton, Section Leader, Specialty Engineering - ISI M. Muhs, Department Leader, Mechanical/RAMS G. Overbeck, Vice President, Nuclear Production T. Radke, Director, Outages M. Radspinner, Section Leader, Design Engineering D. Smith, Director, Operations M. Winsor, Department Leader, System Engineering INSPECTION PROCEDURES USED 37551 61726 62707 71707 71750 92901 92902 92903 Onsite Engineering Surveillance Observations Maintenance Observations Plant Operations Plant Support Activities Plant Operations Follow-up Maintenance Follow-up Engineering Follow-up
't
,j P j
0
-2-ITEMS OPENED AND CLOSED
~Oened 50-528,50-529,50-530/9814-06 VIO Five Examples of Failure to Meet TS 3.5.2 (Section 08.1)
50-528,50-529,50-530/9814-07 50-528,50-529,50-530/9814-08 VIO Two Examples of HPSI Pump Discharge Check Valves Conditions Adverse to Quality (Section 08.1)
VIO Two Examples of Inadequate HPSI Discharge Check Valve Procedures (Section 08.1)
50-528,50-529,50-530/9810-01 50-528;50-529,50-530/9810-02 50-528;50-529;50-530/9810-03 NCV NCV NCV Failure to establish adequate measures to ensure correct parts are selected for modifications (Section M7.1)
Failure to establish adequate measures to ensure that nondestructive testing is accomplished (Section M7.1).
Failure to maintain design control measures for replacement part commensurate with the original design (Section E3.1)
Closed 50-528;50-529;50-530/9814-01 APV Three Examples of An Apparent Violation of TS 3.0.3 (Section 08.2)
50-528;50-529;50-530/9814-02 50- 528;50-529; 50-530/9814-03 APV Two Examples of An Apparent Violation of TS 6.8.1 (Section 08.3)
APV Five Examples of APV of 10CFR Part 50, Appendix B Criterion XVI (Section 08.4)
50-528;50-529;50-530/9814-05 50-528/98-002 APV Two Examples of APV of TS 3.5.2 (Section 08.5)
LER Reactor Trip on Low Steam Generator Level due to Insufficient Feedwater Flow Followed By a Auxiliary Feedwater Actuation Signal (Section 08.6)
50-528;50-529;50-530/9810-01 50-528;50-529;50-530/9810-02 NCV NCV Failure to establish adequate measures to ensure correct parts are selected for modifications (Section M7.1)
Failure to establish adequate measures to ensure that nondestructive testing is accomplished (Section M7.1)
i"I I
-3-50-528/9803-03 50-528;50-529;50-530/9810-03 50-529/9715-03 VIO EDG Internal Inspections Performed Without Proper Authorization (Section M8.1)
NCV Failure to maintain design control measures for replacement part commensurate with the original design (Section E3.1)
. Failure to Follow Procedures (Section E8.1)
I
-4-LIST OF ACRONYMS USED AO APV ASME CEA CFR CRDR CRS DMWO EDG ESP HPSI IA Ib/in LER NRC NCV PDR pslg SIMS TS Vac VIO WO auxiliary operator apparent violation American Society of Mechanical Engineers control element assembly Code of Federal Regulations condition report/disposition request control room supervisor design master work order emergency diesel generator essential spray pond high pressure safety injection Instrument Air pounds per inch licensee event report Nuclear Regulatory Commission noncited violation Public Document Room pounds per square inch station information management system Technical Specification volts alternating current violation work order
-5-LIST OF DOCUMENTS REVIEWED FOR SECTION M7.1 Procedures Number 30DP-OAP01 30DP-9MP01 30DP-9WP02 63DP-OQQ06 73DP-9WP04 73DP-OZZ07 81DP-ODC16 90DP-OIP10 CRDRs Description Maintenance Instruction Writer's Guide Conduct of Maintenance Work Document Development and Control Revision Revision 22 Revisions 19-25 Revision 25 Welding and Brazing Control Welding of Stainless and Nickel Alloys Engineering Document Change Condition Reporting Revision 2 Revision 6 Revision 6 Revision 4 Verification and Inspection of Maintenance Activities Revision 21 9800116 9800289 9800367 9800424 9800425 9800515 9800613 9800819 9800871 9800919 9800924 9800937 9800938 9800939 9800940 9800947 9800966 9800971 9801179 Work Orders 717992 717994 737059 741775 746122 765532 767819 767831 767833 768173 768391 788630 804325 804327 812295 830083
f I
I
I I