IR 05000269/1997006

AIT Insp Repts 50-269/97-06,50-270/97-06 & 50-287/97-06 on 970503-09.Major Areas Inspected:Operations,Engineering, Plant Support & Event Assessment Re 970503 Event Associated W/Degradation of Unit 3 HPI Sys During Plant Cooldown

ML15118A214
Person / Time
Site:	Oconee
Issue date:	05/30/1997
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II)
To:
Shared Package
ML15118A213	List: IR 05000269/1997006 ML15118A212
References
50-269-97-06, 50-269-97-6, 50-270-97-06, 50-270-97-6, 50-287-97-06, 50-287-97-6, FACA, NUDOCS 9706090188
Download: ML15118A214 (39)

Text

U.S. NUCLEAR REGULATORY COMMISSION

REGION II

AUGMENTED INSPECTION TEAM (AIT) INSPECTION Docket Nos.:

50-269, 50-270, 50-287 License Nos.:

DPR-38, DPR-47, DPR-55 Report No.:

50-269/97-06, 50-270/97-06, 50-287/97-06 Licensee:

Duke Power Company Facility:

Oconee Nuclear Station, Units 1, 2 & 3 Location:

P. 0. Box 1439 Seneca, SC 29679 Dates:

May 3 - 9, 1997 Team Leader:

W. Holland, Chief, Maintenance Branch, DRS Inspectors:

T. Cooper, Resident Inspec

, Crystal River G. Humphrey, Re et I cto Oconee J. Ka ff an, Se lor e or S t ms Engineer, AEOD Approved by:

/

o s P. Jau on, irector Date Signed ivision of Reacto fety 9706090188 970530 PDR ADOCK 05000269 G

PDR

SUMMARY Oconee Nuclear Station, Units 1, 2 & 3 NRC Inspection Report 50-269/97-06, 50-270/97-06, 50-287/97-06 The Augmented Inspection Team (AIT) reviewed the facts surrounding an event associated with degradation of the ONS Unit 3 high pressure injection (HPI) system during a plant cooldown on May 3, 1997, the licensee's response to the event and licensee activity during their event, review and recovery. In addition, the AIT assessed generic aspects of ONS operations/inspections to evaluate applicability of the event to the other units. The report covers a one-week period of reactive inspection using a team leader, two resident inspectors, and a senior reactor systems engineer from the NRC Office for Analysis and Evaluation of Operational Dat Operations

The AIT's review of operational information and sequence of events agreed with the licensee's sequence of events. The AIT noted that Letdown Storage Tank level indicated 55.9" for approximately 1-1/2 hours during the plant cooldown, which was abnormal, (Section 01.1).

• The licensee's actions, after determining that potential damage had occurred to two of the three high pressure injection pumps, was considered good. The licensee mobilized emergency response facilities to provide additional technical and management support and developed specific recovery procedures to address system degradation conditions, (Section 01.1).

• The AIT review of this specific event scenario concluded that the off-normal procedures provided guidance for realignment of high pressure pump suction to the borated water storage tank if operators had recognized the letdown storage tank level indication was not accurate. The abnormal procedure appeared adequate to respond to symptoms if letdown storage tank level indications had been accurate; however, it provided limited assistance for this event, (Section 03.1).

• The shutdown/cooldown procedure did not provide guidance for appropriate sensitivity to reactor coolant system and associated systems inventory balancing during cooldown and allowed for operation with letdown storage tank level in the alarm condition, which were considered weaknesses, (Section 03.1).

• The AIT concluded the guidance provided by the Operations Management Procedure may have provided a mixed message to the operators. The statement that procedures may not cover every event and operator deviation from the procedures may be necessary had the potential to give operators the message that the procedures were weak and procedure compliance was not required for events or other operating activities, (Section 03.1).

• At the time of the event, plant cooldown was in progress, which required focused vigilance of inventory balancing parameters. This was not being accomplishe Although the AIT viewed this issue as a crew problem, the AIT also noted the operator-at-the-controls was given the duty as the "dedicated" Low Temperature/Over pressure operator. Limited procedural guidance, coupled with operator knowledge and performance weaknesses prior to the event, contributed to the initiation of the event and potential damage to two HPI pumps. Also, the AIT noted operator responses to the off-normal indications appeared more knowledge-based than rule based (Section 04.1).

Maintenance

Maintenance practices for the calibration of the Letdown Storage Tank level instruments rendered the one of the two level instruments inoperable at a tim Operations logging practices for the removal of these Technical Specification instruments for calibration during operation was weak, (Section M1.1).

• The licensee had created procedures to deal with past problems identified at the site relating to compression fitting maintenance. Implementation of these procedures was not being adequately performed. In addition, adequate maintenance practices for the removal and reinstallation of test caps were not being performed, nor were the appropriate procedures included as references in work packages used to perform calibrations on the instruments. The mixing of components from differing manufacturers was contrary to the licensee's procedures. The failure to notify engineering when mismatched components were identified was contrary to the licensee's procedures, (Section M1.2).

• Six out of 12 letdown storage tank instrumentation line root valves were identified with labelling tags that were different than the valve identifications on instrumentation drawings. Also, the AIT concluded the lack of documentation for initial valve positioning to be a weakness in the licensee's process for valve configuration control, (Section M1.3).

Engineering

The AIT concluded the single reference leg design provided vulnerability to common mode failure for letdown storage tank level indication which is relied upon for high pressure injection pump operability. Also, a continuous fill line which was installed in the plant as part of the level instrumentation lines was not valved into service. The licensee did not consider the continuous fill line to be part of the instrumentation design. The licensee identified vulnerabilities associated with the design of HPI pump piping (a common suction to all pumps) and the letdown storage tank level and pressure instrumentation as early as 1980. The licensee had proposed modifications to address these vulnerabilities but had not implemented them. The AIT concluded the proposed modifications, if successfully implemented, would likely have prevented this event, (Section E1.1).

• The licensee had plans to disassemble the degraded pump internals at a later dat The AIT reviewed the licensee's procedure for pump disassembly and discussed the process of damage assessment with engineering personnel. The damage assessment plans appeared reasonable, (Section E1.2).

• The licensee's event investigations using a failure investigation process team and a significant event investigation team was considered to be generally thorough and comprehensive, with some exceptions noted by the AIT, (Section E7.1).

• Exceptions noted to the licensee's event investigation team's findings were: (1) the licensee's inadequate reviews of past industry events for potential generic problem areas and a lack of implementation of corrective actions to prevent similar occurrences at ONS was a contributor to the May 3, 1997, event; (2) operator interviews by the licensee event teams could have been more beneficial if they had been conducted individually or individual statements obtained; and (3) maintenance processes allowed for work packages which did not contain references or requirements to use procedures, (Section E7.1).

Plant Support

The licensee's performance related to notifications and reporting (i.e., classifying the event, making offsite notifications, on-site response, and interface with offsite emergency agencies was good (Section P1.1).

• One exception relating to reporting was noted regarding the timeliness of reporting as required by 10 CFR 50.72, (Section P1.1).

Event Assessment

The AIT determined the root cause of the event was inappropriate use of plant and industry operating experience to assure that plant design, maintenance, and operation were focused on reliable operation of the HPI System/Components involved in this event, (Section A 1.1).

II REPORT DETAILS Augmented Inspection Team Charter On May 5,1997, an AIT was established by the NRC Region II Administrator to inspect and assess the facts surrounding an event that resulted in degradation of the ONS Unit 3 high pressure injection system during a plant cooldown on May 3, 1997. The Augmented Inspection Team Charter Memorandum, with Attachment, is included as Attachment A of this repor Summary of Plant Status Prior to the event on May 3, 1997, ONS Unit 1 was operating at full power, ONS Unit 2 was in cold shutdown with reactor vessel water level being maintained at approximately 17 inches, reduced inventory (top of reactor coolant loop piping), and ONS Unit 3 was being cooled down in order to conduct inspections of high pressure injection piping at the interface of the reactor nozzl I. Operations

Conduct of Operations

'0 Unit 3 High Pressure injection System Everit

. Inspection Scope (93800)

The AIT Charter required the AIT to, "Assess the licensee's activities related to event recovery (i.e., actions to recover plant systems and establish contingencies to restore plant cooldown)." The AIT interviewed personnel and documentation associated with the loss of HPI during shutdown and cooldown of Unit 3 that occurred from May 2-4, 1997. The documentation consisted of the Reactor Operator's Shift Log, SPOC Manager Log, Technical Support Center Manager Log, and other documentation as note The AIT Charter required the AIT to "Develop a sequence of events associated with the degradation of the ONS Unit 3 HPI system during a plant cooldown on May 3, 1997." The AIT performed a review of operator logs, licensee interviews, maintenance logs, TSC logs, OSC logs, strip charts, and computer logs to develop a sequence of events for the loss of two HPI pumps during cooldown of Unit The AIT interviewed the Unit 3 operators that were on shift at the time the event occurre Observations and Findings SYSTEM DESCRIPTION - LETDOWN STORAGE TANK A Summary Flow Diagram of the High Pressure Injection System is included as Attachment B to this report. A "Letdown Storage Tank Pressure verses Indicated

Level" curve is included as Attachment C to this report. A Letdown Storage Tank Level Instrumentation Sketch is included as Attachment D to this repor There is one Letdown Storage Tank per unit. During normal operation, the tank receives water from the reactor coolant system flowpath and the reactor coolant pumps seal return flowpath (Attachment B). The contents of the tank are used to supply the high pressure injection pumps with water to makeup to the reactor coolant system and to inject into the reactor coolant pumps seal cavities. Hydrogen over pressure is normally maintained on the letdown storage tank to assist in removing dissolved oxygen from the reactor coolant syste In an emergency situation (Engineered Safeguards Actuation), the suction source for the high pressure injection pumps is automatically diverted from the letdown storage tank to the borated water storage tank when engineered safeguards Valves HP-24 and HP-25 (isolation valves between borated water storage tank and high pressure injection pumps suction header) open. Initially the head pressure resulting from the higher borated water storage tank elevation overcomes the combined elevation head and gas pressure of the letdown storage tank, thereby ensuring the borated water storage tank is the suction source for the high pressure injection pumps. As the accident progresses, and the borated water storage tank water inventory is reduced, the elevation head of the borated water storage tank water will decrease such that the letdown storage tank water level-will begin to decrease. This occurrence is expected and creates no problem as long as the letdown storage tank water level does not decrease to the extent that the hydrogen gas expands into the high pressure injection pump suction piping. Accordingly, a "Letdown Storage Tank Pressure verses Indicated Level" curve (Attachment C), is contained in plant operating procedures to define letdown storage tank inventory operating parameters. Adherence to this curve is required to ensure that hydrogen entrainment within the high pressure injection pumps suction piping will not occur prior to entering the low pressure injection/high pressure injection piggyback mode of operation for the accident scenari At the time of the event, the LDST had been vented to approximately atmospheric pressure, with the high pressure injection pump alignments as shown in Attachment LICENSEE ACTIVITY RELATED TO EVENT RECOVERY On May 3, 1997, at 9:13 a.m., the Unit 3 reactor operator noted in the shift log that panel alarm, 2SA-2/C-2 "HPIP DISCH PRESSURE LOW," annunciated and then immediately cleared. The alarm then reflashed within a few minutes indicating that HPI pump discharge pressure was approximately 2375 psig. (normal HPI pump discharge pressure should be approximately 3100 psig). The operator then placed the RCP SEAL FLOW CONTROL VALVE (3HP-31), in the manual position to settle out "swings" noted with the position indication. Low seal pressure alarms for all RCPs annunciated when the pressure dropped to 2375 psig and the Standby Pump, 3A HPI pump auto started. The seal flows and pressure returned to normal, and the operator secured 3A HPI pump. The operator began adjusting Valve 3HP-31, and the seal injection pressure then decreased to approximately 1000 psig. At that time the 3A HPI pump restarted with amps swinging above normal (50 amps) to a range of 70 120 amps. The 3B HPI pump was indicating 10 amps and was shut down. Seal

injection flows dropped from their normal total flow rate of approximately 40 gpm to less than the alarm set-point of 24 gpm. The operator then opened the 3A HPI BWST SUCTION VALVE (3HP-24), to provide suction to the 3A HPI pump from the BWS However, the seal injection flow chart was indicating a zero flow, and the 3A HPI pump was shut down. At this point, both the 3A and 3B HIP pumps were considered degrade Abnormal Procedure, AP/3/A/1700/14, "LOSS OF NORMAL HPI MAKEUP AND LETDOWN," Revision 1 was entered and applicable steps as determined by the SROs and the assistant operations manager were completed. The 3C HPI pump was the only remaining HPI pump that had not failed at that time and the operators discussed starting it. However, the decision was made to not start the 3C HPI pump based on the possibility of damaging it, the pump alignment would not inject into the operating RCP seals, and the integrity of the reactor nozzle that this pump would inject into was questionable. (Note: problems were suspected with the weld and thermal sleeve at the reactor nozzle). Additionally, the RCS cooling and inventory did not present a problem at that time and other abnormal procedure actions were not implemented because the operating crew determined that the procedure was not designed for the existing plant condition At this point (May 3, 1997, approximately 10:30 a.m.), the licensee took prompt actions to staff the Technical Support Center (TSC) and Operations Support Center (OSC) to provide expertise and resources to support the plant recovery process. The TSC directed actions included development of a new procedure to re-align the 3C HPI pump from the BWST and supply makeup to the reactor coolant system (RCS). In addition, contingency procedures were generated in the event the expected results were not obtained when the 3C HPI pump was started. The new procedure and contingencies were completed at approximately 10:30 a.m., on May 4, 1997, and the 3C HPI pump was started. However, there was no pump discharge pressure indicated in the control room (CR) and the pump was stopped. Local monitoring of the 3C HPI pump discharge pressure was then initiated and the pump was restarted with acceptable flow and pressur The resident inspectors reported to the site to monitor the licensee's activities and plant conditions until the event was terminated. NRC Region II activated the Emergency Response Center, dispatched a manager to the site, and maintained constant communications with the site until the plant conditions were returned to normal (3C HPI pump injecting coolant into the RCS and plant cooldown was progressing as expected.)

SEQUENCE OF EVENTS The following is a sequence of events developed by the AIT using plant data and licensee information relating to their investigation of the even February 22, 1997 Unit 3 Letdown Storage Tank (LDST) instrumentation calibration complete Reference leg verified ful May 1. 1997 7:46 pm Started Unit 3 power reduction to cold shutdown using OP/3/A/1 102/10,

"CONTROLLING PROCEDURE FOR UNIT SHUTDOWN," Revision 151 to inspect Unit 3 HPI nozzle thermal sleeve May 2, 1997 4:43 am Main turbine tripped as part of reactor shutdow :00 am Pressurizer level decreased from approximately 240 inches to approximately 220 inches as part of power reductio :55 pm Began RCS cooldown to cold shutdown. RCS temperature-is approximately 5250F 4 to 5 pm 1000 gallons of coolant added to LDST (2 adds). RCS temperature decreases to approximately 505'F 5 to 6 pm 2845 gallons of coolant added to LDST (1 add). RCS temperature decreases to approximately 480'F 6 to 7 pm No adds were made. RCS temperature decreases to approximately 460 0 to 8 pm 2712 gallons of coolant added to LDST (2 adds). RCS temperature decreases to approximately 430'F 8 to 9 pm 2830 gallons of coolant added to LDST (2 adds). RCS temperature decreases to approximately 390aF 9 to 10 pm 3250 gallons of coolant added to LDST (2 adds). RCS temperature decreases to approximately 3650F 10 to 11 pm 1368 gallons of coolant added to LDST (1 add). RCS temperature decreases to approximately 340aF 11:58 pm LTOP established with dedicated operator. RCS temperature is approximately 3350F. Pressurizer level reduced to approximately 100 inches for cooldow May 3, 1997 1:00 am LDST H2 vented. RCS temperature is approximately 3000F 1:19 am 1540 gallons of coolant added to LDST (1 add)

2:40 am RCS < 250aF and 350 psig, 3B HPI pump running, 3A HPI pump in standby 3:00 am RCS temperature is approximately 2400F. Pressurizer level stabilized at approximately 100 inche :00 am RCS pressure 272 psig, temperature 2360F 7:45 am LDST level indicates 55.9 inches and is no longer decreasing - this indication exists for approximately 1.75 hour8.680556e-4 days <br />0.0208 hours <br />1.240079e-4 weeks <br />2.85375e-5 months <br /> :00 am RCS temperature is approximately 2250F. Cooldown has resumed following shift turnove :00 am RCS temperature is approximately 2050F 9:07 am Operators secure 3A2 RCP (One RCP left running)

9:12 am The licensee estimates that the LDST and HPI pump suction piping were empty at this tim :13 am HPIP DISCHARGE HEADER PRESSURE LOW statalarm received and cleared. Reactor Operator notes RCS makeup flow norma :14 am HPIP DISCHARGE HEADER PRESSURE LOW statalarm was received a second time; 3HP-31 observed to be cycling and RCP TOTAL SEAL FLOW LO statalarm received. Pump discharge pressure fluctuating between normal and low pressur :15 am HPI pump 3A auto starts from a low seal injection flow signal. Operator shuts HPI pump 3A down as seal injection flow appears to be norma :16 am Operator places HPI pump 3A in auto and pump restarts. Operators observed high amps on 3A HPI pump motor and low amps on 3B HPI pump motor. HPIP DISCHARGE HEADER PRESSURE LOW and RCP TOTAL SEAL INJECTION FLOW LOW alarms are both receive :17 am Operator shuts down 3B HPI pump. 3A HPI pump amps, at 70-120, were running higher than 3B HPI pump amps, at 10 (50 is normal).

The operators thought that the 3A HPI pump was pumping water, while 3B HPI pump was not. Valve 3HP-31 was shut by operator :21 am Operator opens 3HP-24, providing BWST suction source for the HPI pumps. Total seal injection flow spikes, but then returns to lo :22 am RCP seal injection flow goes to zero and remains ther :28 am Operator closes 3HP-24, isolating BWST suction path. (The AIT could not determine why this flowpath was secured at this time other than operators did not notice any improvement in system response during this period.)

9:30 am 1603 gallons of coolant added to the LDST (1 add)

9:31 am Operator shuts down 3A HPI pump. Operators noted that 3A HPI pump motor was running at approximately 10 amps, while 3HP-24 was ope Non-licensed operators dispatched to the pump area detect a burning odo :32 am Operators close 3HP-5, isolating RCS letdown. Operators enter AP/1700/14, "LOSS OF HPI MAKEUP."

10:30 am Exited AP/1700/14. The licensee decided not to supply RCP seals from the SSF RC makeup syste :04 am OSC manned 11:10 am TSC manned 11:24 am EOF operational 3:04 pm The licensee declared an Unusual Event 3:15 pm The licensee identified that the reference leg for the LDST level instrumentation was empty; providing a signal error of approximately 55 inches, non-conservatively. I&E completed IP/0/B/202/1F, High Pressure Injection Letdown Storage Tank Instrument Calibration. The level instrument reference leg was refilled by this activit :47 pm NRC notified of the event 3:55 pm EOF released 4:32 pm 3A HPI pump isolated May 4. 1997 3:12 am The licensee verified that the LDST level instrumentation reference leg continuous fill valves on ONS Units 1, 2 and 3 were closed

11:23 am Started 3C HPI pump 11:24 am Operators stopped 3C HPI pump; no discharge pressure indicated on control gauge. Discovered that gauge isolated in system alignmen (Isolated by closure of 3HP-128 to prevent debris from entering RC pump seals)

11:34 am Operators started 3C HPI pump; monitoring local discharge pressure gauge - Cooldown recommenced May 5. 1997 7:43 pm Cooldown requiring 3C HPI pump complete 7:46 pm Licensee terminated the Unusual Event The AIT reviewed the operational information and event sequence and noted the most significant abnormality was a lack of operator sensitivity to inventory balancing during the unit cooldown between 7:45 a.m., and 9:13 a.m., on May 3, 1997. Operators should have acted on the need for reactor makeup during a cooldown based on both their training and experience. The sequence of events between 4:00 p.m., and 11:00 p.m., on May 2, 1997, clearly showed the relationship between plant cooldown and inventory makeup requirements for the LDS The licensee documented the issues discussed in this section in problem investigation process (PIP) 3-097-1428 and 3-097-145 Conclusions The AIT's review of operational information and sequence of events agreed with the licensee sequence of events. The AIT noted that LDST level indicated 55.9" for approximately 1-1/2 hours during the plant cooldown, which was abnorma The licensee's actions, after determining that potential damage had occurred to two of the three HPI pumps was considered good. The licensee mobilized emergency response facilities to provide additional technical and management support, and developed specific recovery procedures to address system degradation condition Operations Procedures and Documentation 0 Availability of Procedures Addressing Contingencies for Such Problems Inspection Scope (93800)

The AIT Charter required the AIT to "Assess the generic aspects of related operations/inspections of all three ONS Units as it relates to: Availability of procedures addressing contingencies for such problems." The AIT interviewed the Unit 3 operators that were on shift at the time the event occurred. The AIT reviewed

operator logs, TSC logs, alarm response procedures, abnormal operating procedures applicable to the loss of HPI, the licensee's Operations Management Procedure (OMP), and the controlling procedure for unit shutdow Observations and Findings The AIT reviewed OP/3/A/1 102/10, "CONTROLLING PROCEDURE FOR UNIT SHUTDOWN," Revision 151 that was utilized to bring the unit from power operation to a cold shutdown condition. The AIT noted that time and dates were not required for major steps completed. in the procedure and neither were the completion of major steps documented in the operator's log. Step 2.0, Limits and Precautions, provided critical parameters for the operators to monitor during the shutdown. However, the AIT found no reference in this step that alerted the operators to the need for appropriate inventory balancing during cooldown evolution During the interview with the operators, the AIT leamed that the operators had responded initially to the event prior to consulting the Alarm Response Guides (ARG)

or the Abnormal Procedure (AP). Procedures were later utilized as the event progressed toward further plant degradation. However,: the operators stated that the ARGs for the loss of Reactor Coolant Pump (RCP) seal flows were of very little, if any, help to them during the event, due to the unknown LDST level indication proble The AIT reviewed the alarm responses procedures for the following alarms and determined that guidance was provided directing operators to check for low LDST level; however, since the LDST indication was wrong, the guidance was of little or no benefit to the mitigation of pump damag o INJECTION PUMP DISCH HEADER PRESSURE LOW. The alarm was set to annunciate when the pressure dropped to 2375 psig. The required operator actions were:

Verify the HPI header flow does not exceed 475 gpm/pump Start the standby HPI pump as necessary Monitor 'A' and 'B' HPI pumps for proper operation Monitor for low LDST level and Makeup as required by OP/3/A/1103/04 Verify LDST levels as read by the OAC 03A1042 HP LDST LVL 1 03A1043 HP LDST LVL 2 If an accurate LDST level cannot be maintained or verified then align suction for HPI pumps from BWST per AP/3/A/1700/14, (LOSS OF NORMAL HPI MAKEUP AND LETDOWN).

Monitor Auxiliary Building for HPI System Leakage Check for line failure upstream of 3FT-7B in HPI Room Check pump amps to verify pumps running

o RC PUMP 3A1 (i.e., 3A2,3 B1, 3B2) SEAL INLET FLOW LOW. The alarm was set to annunciate when the seal inlet flow drops from its normal range of approximately 10 gpm to 4 gpm. The required operator actions were:

Ensure at least one HPI pump supplying seal injection

Refer to AP/3/A/1700/16, ABNORMAL RCP OPERATION

Check RC pump seal injection flow gauge and adjust 3HP-64 (RCP

"3A1" Seal Injection Throttle), if necessary. Note: 3HP-65, 66, & 67 for RCPs 3A2, 3B1, & 3B2, respectivel Abnormal Procedure, AP/3/A/1700/14, "LOSS OF NORMAL HPI MAKEUP AND LETDOWN," Revision 1, was also determined to be of little benefit to the operators for the condition of the plant at the time of the event. The AP appeared adequate to respond to the symptoms if letdown storage tank level indications had been accurat The operators exited the AP and maintained the plant in a stable and safe condition while waiting for the TSC to devise a recovery plan. During the next 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />, the licensee generated an enclosure to Operations Procedure OP/3/A/1 104/02, "HIGH PRESSURE INJECTION SYSTEM," Revision 84, Enclosure 5.17A, "OPERATION OF 3C HPI PUMP WITH 3A AND 3B HPI PUMP INOPERABLE," and contingencies for establishing and operating the 3C HPI pump with a flow path from the BWST to the RCS. The licensee, in the procedure revision, failed to recognize that there would be no HPI pump discharge pressure indication in the CR due to the required system alignment. This procedure problem resulted in operators stopping the 3C HPI pump immediately after it was started. This was a good, conservative action. The pressure indication problem was resolved by the operations staff and the pump restarte Step 2.2 of the Limits and Precautions statement in OP/3/A/1 104/02 required that the LDST level be maintained at greater than 40 inches and specified an operating curve based on pressure and level parameters. During interviews, the AIT was informed by the operators that a minimum level of greater than 40 inches was acceptable for the plant conditions at the time of the event. However, the AIT noted that operation at less than 55 inches in the LDST would be below the alarm set-point for low-level, and, therefore, the procedure allowed the LDST level to be maintained in a range lower than the alarm setpoint, which was considered a weaknes The AIT reviewed the operator's actions in deviating from the AP during the event and determined the deviations were allowed per Operations Management Procedure (OMP) 1-9, USE OF PROCEDURES, Revision 25. The OMP provided guidance for procedure deviation when a procedure was determined to be deficient or not applicable for the condition experienced to place the plant in a safe condition. The OMP stated "during off-normal plant operations, emergency and abnormal procedures are provided to the operator to respond to the events. Operators are expected to follow the procedures when responding to off-normal events. However, certain situations may arise where the guidance provided by the emergency or abnormal procedure is deficient or not applicable. In these cases, the operators may deviate from their procedures as necessary to place the plant in a safe state. Such deviations shall be approved by the operations shift manager or, in his absence, the unit shift supervisor or control room SRO." Also, the OMP general statement of philosophy

stated that, "Procedures do not cover all situations. Qualified operators are required to take appropriate action to place the plant in a safe condition, independent of procedures."

The AIT discussed this philosophy with licensee management. The AIT considered that operators may be receiving a mixed message based on this guidance. The statement that procedures may not cover every event and operator deviation from the procedures may be necessary had the potential to give operators the message that the procedures were weak and procedure compliance was not required for events or other operating activities. Licensee management was receptive to reviewing this are Conclusions The AIT review of this specific event scenario concluded the off-normal procedures provided guidance for realignment of high pressure pump suction to the borated water storage tank if operators had recognized the letdown storage tank level indication was not accurate. The abnormal procedure appeared adequate to respond to symptoms if letdown storage tank level indications had been accurate; however, it provided limited assistance for this event. The shutdown/cooldown procedure did not provide guidance for appropriate sensitivity to reactor coolant system and associated systems inventory balancing -during cooldown and allowed for operation with letdown storage tank level in the alarm condition which were considered weaknesse The AIT concluded the guidance provided by the Operations Management Procedure may have provided a mixed message to the operators. The statement that procedures may not cover every event and operator deviation from the procedures may be necessary had the potential to give operators the message that the procedures were weak and procedure compliance was not required for events or other operating activitie The licensee documented the issues discussed in this section in PIP 0-097-1455 and 0-097-146 Operator Knowledge and Performance 0 Operator Performance During the Event Inspection Scope (93800)

The AIT reviewed operator logs, licensee interviews, maintenance logs, TSC logs, OSC logs, strip charts, and computer logs; and developed sequence of events for the loss of two HPI pumps during cooldown of Unit 3. In addition, the AIT conducted operator interviews with three SROs and three ROs, who were on shift during the event (7:00 a.m.- 7:00 p.m., May 3, 1997).

b. Observations and Findings MONITORING DURING COOLDOWN With a plant cooldown in progress, the crew did not maintain a focused vigilance on inventory balancing parameters. This was a fundamental crew performance weakness, in that the crew did not monitor key parameters closely for the expected response; and hence the operators did not observe, identify, and question an anomalous plant response. Specifically, operators did not observe and question the constant LDST level that existed for about 1.5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> prior to the event. Operators should have understood that with a cooldown in progress, LDST level would decrease due to shrinking of the primary coolant (with pressurizer level being maintained constant), necessitating makeup to the LDST. Interviews with the operators on shift at the time of the event, revealed that the operator at the controls, who was in charge of the overall operations of the unit, was concurrently the dedicated LTOP operato These concurrent responsibilities had the potential of affecting that individual's performanc Procedural guidance available (OP/3/A/1 102/10, "CONTROLLING PROCEDURE FOR UNIT SHUTDOWN," Revision 151) to the operators for the cooldown, however, did not greatly assist the operators. Procedural guidance did not sufficiently caution or alert the operators to the importance of monitoring key inventory balancing parameters or to the expected responses of key parameters during a cooldown. Such guidance, could have, for example, required more frequent logging or trending of key parameters. If the operators had identified and questioned the constant indicated LDST level, it was likely they would have either stopped the cooldown, initiated makeup to the LDST, or investigated and identified the level indication problems. Any of these actions would likely have led to resolution of the problem without any potential damage to the HPI pump INITIAL OPERATOR RESPONSE Once HPI pump alarms were received, operators were somewhat misled in that the alarms could have been related to the recent securing of an RCP and related seal injection flow balancing. When HPI pump 3A automatically started on a low RCP seal injection flow signal, the operators initially reacted with a knowledge-based, rather than a rule-based (proceduralized) response. The operators shutdown HPI pump 3A within 39 seconds after verifying RCP seal injection flow appeared adequate. About 30 seconds later, operators returned HPI pump 3A to automatic, and it automatically restarted. Based on interviews, operators checked the LDST level indicators and swapped the level recorder to the other channel. All level instruments and the level recorder (either input instrument selected) agreed and read above the alarm setpoint, indicating to the operators that LDST level was not a problem. (The exact timing of these LDST level checks is unclear; the checks may have actually occurred later in the event.) Operators checked pressurizer level and found it stable. Because LDST level appeared normal, operators hypothesized the problem was either an HPI line break or a problem with the suction sourc At 9:17 a.m., operators secured HPI pump 3B. This action was based on a belief that HPI pump 3A was pumping water because its motor amps were higher than those of HPI pump 3B. HPI pump 3A motor amps indicated between 70 and 120, whereas normal running amperage is about 50 amps. Over about the next 13 minutes, operators received numerous alarms concerning letdown flow and RCP seal injection flow. At 9:22 a.m., operators aligned HPI pump suction from the BWST. At 9:28 a.m., operators secured HPI pump suction from the BWST. At 9:31 a.m.,

operators secured HPI pump 3A after non-licensed operators reported smoke from the HPI pump room. Operators observed that HPI pump 3A motor amps were about 10 amps while HPI pump suction aligned from BWST. At 9:32 a.m., operators isolated letdown and entered AP/3/A/1700/14, "LOSS OF NORMAL HPI MAKEUP AND LETDOWN" abnormal procedure. Operators also entered AP/3/A/1700/16,

"ABNORMAL REACTOR COOLANT PUMP OPERATION," Revision Based on interviews, the operators on duty erroneously believed that the LDST level indicators were independent. As such, the operators were not aware that a problem with a reference leg, as apparently occurred in this event, could affect both instruments. The operators' belief that the LDST level instruments were independent was reinforced by performance of routine channel checks on these instruments. None of the operators interviewed could recall any training regarding the lack of independence of the LDST level instruments. Operator unawareness of this lack of independence may have contributed to their not identifying or questioning the loss of water level in the LDST as a potential problem during their initial response. However, even if operators had been trained and aware of the lack of independence, it may have been unreasonable to expect operators to recall this level of detail while responding to alarms and system transients, although labelling (or similar operator aids) could alert operators to this lack of independence when they are reading the instrumen HPI pumps 3A and 3B were run about four minutes and 17 minutes, respectively, during this event. During much of this time, abnormal indications existed to alert the operators to abnormal pump or system conditions. Plant conditions did not require-IPI pump in that the cooldown could have been stopped (minimizing the need for high pressure makeup) and reactor coolant pumps seals could be (as was demonstrated during the event) and were coolable by the component cooling water system. The AIT noted that operators did not enter abnormal and alarm response procedures until about 15 minutes into the event. The AIT viewed this as resulting in a knowledge-based rather than a rule-based initial respons The use of procedures in responding was important for a number of reasons, including imperfect operator memory; the tradeoffs among complex and complicated actions could be evaluated prior to the event, and the best or most important actions identified and written down, rather than operators attempting to make difficult choices during an event; and procedures could have allowed for systematic and methodical troubleshooting and response. In this case, the operators used an ad hoc, non systematic approach to responding, which may have contributed to additional HPI pump damage. However, the licensee estimated after the event that HPI pump damage could occur in as little as 20 seconds after HPI pump loses suction. This

estimate underscored the fragility of the pumps and the need for operator understanding and awareness of this fragility when responding to HPI pump problem Based on this estimate, it appeared that providing automatic HPI pump protection (such as low suction pump trips), rather than relying on operator action, should be considered as a potential corrective action. The operator knowledge and performance weaknesses evident before and during this event suggested that either a more proceduralized approach be used or that some operator knowledge and skills be enhance SUBSEQUENT OPERATOR RESPONSE Subsequent operator response, including not starting the third HPI pump, careful monitoring of running reactor coolant pump parameters, and decisions to staff emergency response facilities were conservative. The decision not to start 3C HPI pump was particularly noteworthy, although it was unclear to the AIT whether it was not started because it was not aligned for RCP seal injection, thermal stress concerns with its injection nozzle, or knowledge that with two HPI pumps potentially damaged and suspected problems with the HPI pump suction sources, that extreme care should be exercised before starting it. Operator actions to monitor RCP parameters were likewise conservative. The licensee's decision to staff emergency response facilities to support the on-shift crew was pruden Limited procedural guidance, coupled with operator knowledge and performance weaknesses prior to the event, contributed to the initiation of the event and potential damage to two HPI pumps. Operator response to the event was mixed, in that a non proceduralized, knowledge-based response was used for approximately 15 minutes of the initial response. This response may have contributed to the potential damage to two of the HPI pumps, when these HPI pumps were started or run while problems existed with their suction source. Subsequent operator response, including not starting the third HPI pump, careful monitoring of running reactor coolant pump parameters, and decisions to staff emergency response facilities were conservativ The licensee documented the issues discussed in this section in PIP 3-097-145 Conclusions At the time of the event, plant cooldown was in progress, which required focused vigilance of inventory balancing parameters. This was not being accomplishe Although the AIT viewed this issue as a crew problem, the AIT also noted the operator-at-the-controls was given the duty as the "dedicated" LTOP operato Limited procedural guidance, coupled with operator knowledge and performance weaknesses prior to the event, contributed to the initiation of the event and potential damage to two HPI pumps. Also, the AIT noted operator responses to the off-normal indications appeared more knowledge-based than rule-base II. Maintenance M1 Conduct of Maintenance M Calibration of Letdown Storage Tank Level Instrumentation Inspection Scope (93800)

The AIT reviewed maintenance activities associated with calibration of the letdown storage tank level instrumentatio Observations and Findings The AIT reviewed the process used by the licensee to perform maintenance on the LDST level instrumentation. Two procedures, IP/0/1B/0202/001 F, "HIGH PRESSURE INJECTION SYSTEM LETDOWN STORAGE TANK LEVEL INSTRUMENT CALIBRATION", Revision 29, and IP/O/A/0075/010, "INSTRUMENT IMPULSE LINE FILLING", Revision 2, were used by the licensee to perform calibration of the instruments on a refueling interva The inspector discussed the performance of the procedures with engineering and I&E personnel and accompanied an I&E technician into the field to discuss normal practices while performing the procedures. The inspector verified that the performance of the calibration on one instrument had no effect on the second instrument. During the performance of the reference leg fill, no affects should be noted on the second instrument, unless the reference leg was not full, at which time, both instruments would be impacte The AIT noted that the LDST level instrumentation was included in TS Table 4. This table provided requirements for a daily channel check and a refueling interval calibration but provided no action statement or requirements for operabilit Interviews with control room personnel and reviews of control room logs revealed that the operators did not make a TS log entry when these instruments were removed for calibration. A log entry in the shift log was noted on May 3, 1997, on Unit 1 for the performance of the calibration, but no note was made that these were TS instrument Procedure IP/0/B/0202/001F noted that the instruments were in TS, but did not provide any guidance or requirements for removing them from servic Conclusion Maintenance practices for the calibration of the Letdown Storage Tank level instruments rendered the one of the two level instruments inoperable at a tim Operations logging practices for the removal of these Technical Specification instruments for calibration during operation was wea M Control of Instrumentation Connectors Inspection Scope 193800)

Loss of reference leg inventory resulted in a problem that caused two letdown storage tank level instruments to read at least 55.9 Inches high. The licensee identified that one of the causes for the May 3, 1997, event was a leaking test tee connection on the LDST level instrumentation, allowing the reference leg on the instrument to partially drain. The AIT reviewed the licensee's evaluation and performed field walk-downs and documentation reviews to assess the licensee's efforts in evaluating the leaking test tee issu Observations and Findings The AIT reviewed PIP 3-097-1461, issued on May 7, 1997, which documented that a LDST level transmitter test tee was discovered to have parts installed from different manufacturers. The licensee had discovered that the test tee was manufactured by Swagelok, while the tee cap and isolation plug were manufactured by Parker-Hannifi The licensee had concluded that the reference leg test tee for the Unit 3 LDST level instrument 3HPILT0033P2 had leaked, resulting in partial draining in the instrument reference leg and inaccurate level indications. This was concluded based on the observation of boron encrusted on the outside of the test tee, following the event on May 3, 199 The AIT reviewed documentation for a previous event at the site, which included LER 50-287/91-008, Excessive Reactor Coolant Leak, Reactor Trip, and Inadvertent Protective System Actuation Result From Management Deficiencies and Equipment Failure; and NRC Information Notice 92-15, Failure of Primary System Compression Fitting, which discussed consequences of lack of control of compression fittings in the plant. As part of the corrective actions for the 1991 event, the licensee created IP/0/A/0075/012, "INSTRUMENT LINE TUBE FITTINGS", Revision 0. In May 1996, the licensee created an additional, system-wide procedure; SI/0/A/5090/001, "TUBE FITTING AND TUBING INSTALLATION", Revision Section 7.0, General Description, of IP/0/A/0075/012 stated that the procedure was used to provide guidelines for proper installation of tube fittings of safety-related and non-safety related stainless steel instrument lines ranging from 1/4 to 1 inc Section 5.0, Step 5.6, Limits and Precautions, states: "DO NOT mix or interchange parts from other manufacturers. IF mixed fittings are identified, contact engineering for appropriate action." The AIT spoke with various engineers and engineering management at the site and determined that engineering had not been notified, at any time prior to May 3, 1997, that mixed parts were being used for LDST level instrumentation lines. The failure of the maintenance personnel to assure that parts from different manufacturers were not being used and the lack of a questioning attitude on their part were major contributors to this occurrenc On May 4, 1997, the licensee pressurized the reference leg side test tee cap to 55 psi, per WO 97019502. A leak was noted on 3HPILT0033P2 test cap. On May 5, 1997, the licensee removed the test tee cap for inspection, under WO 97038177. The licensee, in the field, found a scratch on the seating surface of the test tee, but no problems noted on the cap/ferrule. The tee cap was reinstalled and pressure teste On this attempt, the cap began leaking at 45 psi. The cap was removed and reinstalled by a different technician. On this attempt, a leak was observed at 30 ps The cap was again removed and reinstalled by a different technician. On this attempt, the leak occurred higher than 70 psi. The four installations of the cap, by different technicians revealed that a uniform methodology was not in use for installation of compression fittings. Even though a procedure existed with detailed instructions for dealing with compression fittings, it was not being utilized for some components, such as caps, even though it was intended to be applied to these fittings. In addition, the repeated removal and installation of the cap on the test tee suspected of leaking, prior to the final inspection by qualified metallurgical personnel had the potential to have impacted the final evaluatio The metallurgical analysis performed by the licensee concluded that the test tee showed some deformation, indicative of excessive over-tightening and the use of mismatched parts. The threads of the test tee showed signs of non-uniform stretching, indicating cross threading and excessive over-tightening. A circumferential groove 3600 around the bottom of the test tee seating surface showed that the plug had been driven into the seating surface. An "L" shaped scratch on the seating surface of the test tee was evidence that foreign material was compressed between the seating surface of the test tee and plug. The plug had several scratches varying in size and direction, also demonstrating that foreign material was compressed between seating surface of test tee and plug. The metallurgical evaluation also revealed that 65% - 75% of the test tee seating surface showed signs of wear or damage. This was attributed to foreign material being present, the cap being over tightened, and the use of mismatched parts. These observations indicated that proper maintenance practices were not being used during removal and installation of this ca SI/0/A/5090/001, Enclosure 4.12, TUBE CAP INSTALLATION, provided guidance for the proper installation of tube caps but included a note that stated the enclosure was for guidance only and procedure that the procedure did not have to be used as long as the technician was knowledgeable of the practices. The AIT reviewed the last three performances of the instrument calibration; WO 96042013 completed October 21, 1996, WO 96013880 completed February 22, 1997, and WO 97019404, completed on May 3, 1997. Neither SI/0/A/5090/001 nor IP/0/A/0075/012 were referenced by any of the work packages. Even though SI/0/A/5090/001 was only required to be used if the technician was not knowledgeable of the practices, the licensee's work -planning process assumed that all technicians were knowledgeable of the practice. The AIT concluded that since evidence existed, the technicians were neither consistent nor efficient in the removal and installation of the test cap, that they were not knowledgeable of the practice and should have been using the procedural guidanc The licensee documented the issues discussed in this section in PIP 3-097-146 Conclusion The AIT concluded that the licensee had created procedures to deal with past problems identified at the site relating to compression fitting maintenanc Implementation of these procedures was not being adequately performed. In addition, adequate maintenance practices for the removal and reinstallation of test caps were not being performed, nor were the appropriate procedures being included as references on work packages used to perform calibrations on the instruments. The mixing components from differing manufacturers was contrary to the licensee's procedures. The failure to notify engineering when mismatched components were identified was contrary to the licensee's procedure M1.3 Configuration Control of Instrumentation Valves Inspection Scope (93800)

The AIT reviewed the licensee's process for configuration control of valves associated with instrumentation lines at ON Observations and Findings After the event and plant conditions were stabilized on May 3, the licensee performed a verification check of selected valves for Units 1, 2, and 3 to determine the position of valve HPIIV-0086. This valve was identified as the root isolation valve for a continuous fill line as shown on Unit 3 Drawing No. 0-2422X-1, Revision 7. The licensee determined that the referenced valve was closed for each unit. The licensee also noted that several of the instrument root valves for the letdown storage tank were not properly labelled (tagged) as shown on the respective drawings. A resident inspector accompanied the licensee during the valve verifications on Unit 2 and provided the following matrix of labelling deficiencies identified in the following table:

LABEL AND POSITION DRAWING IDENTIFICATION UNIT I UNIT 2 UNIT 3 HPIIV-0086 HPIIV-0085 HPIIV-0085 NOT LABELED CONTINUOUS FILL LINE OPEN CLOSED CLOSED HPIIV-0085 HPIIV-0086 HPIIV-0086 HPIIV-0085 UPPER TAP OPEN OPEN OPEN HPIIV-0087 HPIIV-0087 HPIIV-0087 HPIIV-0087 LOWER TAP OPEN OPEN OPEN

DRAWING IDENTIFICATION UNIT 1 UNIT 2 UNIT 3 HPIIV-0095 PRESSURE PRESSURE PRESSURE PRESSURE TRANSMITTER TRANSMITTER TRANSMITTER TRANSMITTER TAP HPIIV-0086 OPEN Unit 1 had two valves in series located in the continuous fill-line. One valve was unlabelled and closed, the other valve was labelled as note The AIT discussed the observed labelling discrepancies with the licensee. In addition, the AIT requested the licensee to provide documentation as to when the LDST root valves were last verified as in the correct position. The AIT was trying to understand how valve position verifications could be properly performed noting the high number of mislabelled valves (6 out of 12) for the level and pressure instrumentation for the 3 units' respective LDSTs. The licensee could not provide documentation for the last positioning of the valves. However, the licensee provided a copy of Oconee Nuclear Site Directive (ONSD) 1.3.2, "ACTIVITIES AFFECTING STATION OPERATION,"

revised date March 12, 1997. Section 6.4 of ONSD 1.3.4 stated,..."the station maintenance group has operational control of the following station equipment."

Section 6.4.1 stated "All instrument valves including the first valve (Instrument Root Valve) off the process system..." The licensee considered that they had adequate control of these valves and noted that any current operations were specifically controlled by procedures. The AIT did not pursue this item further; however, they considered the lack of documentation for initial valve positioning to be a weakness in the licensee's process for valve configuration contro The licensee documented the issues discussed in this section in PIP 0-097-142 Conclusion Six out of 12 letdown storage tank valves were identified with labelling tags that were

.different than the valve identifications on instrumentation drawings. Also, the AIT concluded the lack of documentation for initial valve positioning to be a weakness in the licensee's process for valve configuration contro III. Engineering El Conduct of Engineering E Single Failure or Common Cause Vulnerabilities of the Letdown Storage Tank Level Indication/Control System Inspection Scope (93800)

The AIT Charter required the AIT to, "Assess the generic aspects of related operations/inspections of all three ONS Units as it relates to: Single failure or common cause vulnerabilities of the letdown storage tank level indication/control system."

b. Observations and Findings The AIT reviewed the design of the LDST level instrumentation. The AIT observed that the LDST instrumentation consists of two level transmitters that feed two level indicators. A control room strip chart recorder could be selected and fed from either of the level transmitters. High and low level alarm relays operated from the strip chart recorder, activated from whichever level transmitter is selected. The design of the instrumentation was such that the reference and variable legs for each level transmitter were common (See ATTACHMENT D). The LDST level instrumentation was originally designed with a continuous fill-line for the reference leg, but the licensee maintained-the. isolation valve in this line closed. The AIT questioned the licensee as to when the continuous fill-line was removed from service. The licensee could not provide any documentation relating to the purpose for the continuous fill-line prior to the AITs departure. The licensee stated that this line was not considered as part of the current design of the letdown storage tank level instrumentation for the plant. Operators used the LDST level indication to maintain LDST level and pressure in an allowable operating band to assure HPI system operability. The indicated LDST level band was from 0 to 100 inches, with about 31 gallons per inch. The only automatic control action activated from low LDST level is that repositioning of the three way valve from its bleed position to its normal positio In this event, a loss of reference leg fill resulted in LDST indicated level remaining (erroneously) normal, while actual LDST inventory was decreasing due to the effects of RCS cooldown. No alarms or off-scale indications were received. This resulted in operation of two HPI pumps with inadequate net positive suction head during this event, with a third HPI pump being vulnerable to the same problem if it had been ru A licensee engineering evaluation of the erroneous LDST level indication problem, on May 5, 1997, determined that the ONS Unit 3 HPI system would not have been able to perform its intended safety function during power operations from February 22, 1997, until the unit was shut down on May 3, 1997. In the event of a small break loss of coolant accident, the potential existed for hydrogen to be drawn into the common suction of the HPI pumps as the borated water storage tank volume was deplete 'Similarly, potentially undetected failures of the LDST level instrumentation could occur, for example, if there was debris or clogging in either the reference or variable leg. Because the HPI pumps at ONS had a common suction line, lack automatic pump protection trips, and receive automatic start signals under certain plant conditions, there was a potential for common mode failure of two or more HPI pumps whenever there was a problem with the HPI pump suction source. Two HPI pumps were needed to meet the success criteria for HPI in ONS accident analyse The licensee had proposed modifications dating back to 1980 to address design vulnerabilities associated with having common suction piping to the HPI pumps. The proposed modifications evolved from installing loss of suction pressure trips on the HPI pumps, to automatically opening the suction valve from the BWST to the HPI pumps under certain conditions, to adding/upgrading the LDST level instruments (up to making them independent, QA-1). In January 1992, ONS Operations provided input to ONS Projects requesting QA-1 LDST level and pressure instruments and interlocking open the BWST supply to the suction of the HPI pumps on either low

LDST level or pressure. In April 1992, ONS Management decided to use existing LDST level and pressure instruments but agreed to add an interlock on LDST pressure to open the BWST supply to the suction of the HPI pumps. In March 1995, a USQ was identified with this modification, delaying its planned implementation to the refueling outage following Cycle 1 The licensee planned to make urgent, minor modifications to Units 2 and 3 in response to this event by May 16, 1997. The urgent modifications would 1) add dual reference legs for the LDST level transmitters; 2) install an electronic pressure measurement device, attached to the top of the existing LDST pressure tap; and 3)

ensure that the low LDST level alarms could be received from either the A or B instrumentation channel The licensee documented the issues discussed in this section in PIP 0-097-142 Conclusions The AIT concluded the single reference leg design provided vulnerability to common mode failure for letdown storage tank level indication which is relied upon for high pressure injection pump operability. Also, a continuous fill-line, which was installed in the plant as part of the level instrumentation lines, was not valved in. The licensee did not consider the continuous fill-line to be part of the instrumentation design. The licensee identified vulnerabilities associated with the design of HPI pump piping (a common suction to all pumps) and the LDST level and pressure instrumentation, as early as 1980. The licensee had proposed modifications to address these vulnerabilities, but had not implemented them. The AIT concluded these proposed modifications, if successfully implemented, would likely have prevented this even E Damage Assessment of High Pressure Iniection Pumps and Their Ability to Function After the Event Inspection Scope (93800)

The AIT Charter required the AIT to "Assess the generic aspects of related operations/inspections of all three ONS Units as it relates to: Damage assessment of the failed HPI pumps and their ability to function after the event." Observations and Findings The licensee briefed the AIT on plans to disassemble, inspect, and perform damage assessment of HPI pumps 3A and 3B. The licensee's overall plan was to: 1) remove HPI pumps 3A and 3B and replace with spare pumps; 2) remove, disassemble, inspect, and replace HPI pump 3A and 3B motors; and 3) assess piping and components downstream of the HPI pumps 3A and 3B for debris, and flush suction piping from the LDST to HPI pumps 3A and 3 While the AIT was onsite, the licensee had removed the 3A and 3B HPI pump motors, and had shipped them to Westinghouse for initial inspection. The licensee had also

removed the 3A HPI pump. Licensee short-term efforts were focused on efforts to rebuild and replace spare HPI pumps, hence detailed disassembly, inspection, and damage assessments were not scheduled to be completed while the AIT was onsit The AIT, however, reviewed maintenance procedures "PUMP - INGERSOLL-RAND HIGH PRESSURE INJECTION -DISASSEMBLY, INSPECTION, and ASSEMBLY,"

MP/O/A/1300/009 (Change 30) and "Pump -Ingersoll-Rand - High Pressure Injection Removal and Replacement of Pump and Motor," MP/O/A/1300/020, Revision 22. The licensee planned to have both mechanics and machinists involved in the pump disassembly, and Engineering expertise was to be available and contacted to evaluate any pump damage which was discovered during the pump disassembly. The licensee planned to contact the pump manufacturer prior to disassembly to ascertain expected pump damag Conclusion The licensee had plans to disassemble the degraded pump internals at a later dat The AIT reviewed the licensee's procedure for pump disassembly, and discussed the process of damage assessment with engineering personnel. The damage assessment plans appeared reasonabl E2 Engineering Support of Facilities and Equipment E Review of UFSAR Commitments While performing inspections discussed in this report, the inspectors reviewed the applicable portions of the UFSAR that related to the areas inspected. For the applicable portions reviewed, the inspectors verified that the UFSAR wording was consistent with the observed plant practices, procedures, and/or parameter E7 Quality Assurance in Engineering Activities E Review of Licensee Event Investigation Team Process Inspection Scope (93800)

The AIT Charter required the AIT to, "Monitor and review the licensee's activities related to the event investigation (e.g., failure of HPI pumps 3A and 3B, root cause analysis, precursor event reviews, etc.). Evaluate the effectiveness of the Failure Investigation Process (FIP) team and the Significant Event Investigation Team (SEIT)

insofar as they complete their tasks within the time period of your inspection." The AIT reviewed licensee activities associated with event review and root cause determination, which included the assessments being performed by the licensee's FIP team and the SEIT. The process used by both teams, along with the conclusions reached were evaluate Observations and Findings On May 3, 1997, after the event occurred, the licensee established a FIP team to review the event, determine its cause, and recommend appropriate corrective action On May 4, 1997, the licensee established a SEIT to review the event and ensure it was investigated in a timely, systematic, and technically sound manner and that a complete understanding of the event and lessons learned was achieve Following the event on May 3, 1997, the licensee performed a review of industry events to identify prior events of a similar nature. The licensee identified that similar events occurred at Arkansas Nuclear One, Unit 2 in 1988, and in 1985 at Palo Verd The AIT identified that several generic communications had been issued by the NRC describing similar occurrences and reference level problems; examples included IN 83-77, which described an event at St. Lucie where a partially drained common reference leg for the Volume Control Tank level instrumentation resulted in inaccurate high level indication, draining of the tank, and impact on operation of the pumps and IN 84-70, which was a generic discussion on problems encountered when using level instruments with common reference leg The AIT reviewed the licensee's disposition of IN 83-77 and IN 84-70. The licensee's past review concluded that IN 83-77, even though it was mainly focused on gas binding of pumps, did include the description of an event very similar to the present event, where a partially drained common reference leg on level instrumentation resulted in inaccurate high level indication on a suction supply for HPI pumps. The licensee concluded that even though similar events could occur at their site, procedures existed to allow for pump venting prior to a pump being returned to service. The AIT considered the review, by the licensee, appeared to have focused on one type of gas intrusion problem, where IN 83-77 clearly stated the events discussed in the IN were intended to be illustrative, and that the types of system inoperability resulting from air or gas entrainment vary. Licensee action for IN 83-77 was training of STAs on the need to vent pumps prior to startin IN 84-70 was a generic concern for both BWRs and PWRs which addressed the problems relating to water level instrumentation with common reference legs. The licensee concluded that level instrumentation on the pressurizer and steam generators share common taps, but no redundant level instruments share either common taps or common reference legs. The AIT concluded the licensee's past review was inadequate, in that the LDST level instruments shared a common reference leg, which was one of the root causes identified for this even The AIT concluded that the licensee's inadequate reviews of past industry events for potential generic problem areas, and a lack of implementation of corrective actions to prevent similar occurrences at ONS was a contributor to the May 3, 1997 even The AIT noted the FIP team and SEIT interviewed the control room operators following the event. No written statements were taken from the operators. In addition, the interviews were conducted in groups; one group of three operators, one group of two operators, and one individual. The AIT interviewed the operators

individually and reached some different conclusions than the licensee investigation teams. The AIT concluded the operators had not used procedures immediately to respond to the event but had responded based on their knowledge. The AIT concluded that the operator interviews could have been more beneficial if they had been conducted separatel The SEIT determined that there were two root causes that resulted in the potential damage to the 3A and 3B HPI pumps. They were:

The design weakness of a common reference leg for the LDST level instruments combined with a leaking instrument fitting that resulted in inaccurate LDST level indicatio *

The failure of the control room crew (operators) to monitor and detect the inaccurate level indications given the existing plant condition The FIP team concluded: the direct cause of the faulty level signals was a loss of water inventory out of the common reference leg for both level channels, the direct cause of the loss of inventory was a leaking test cap connection off the reference leg, the causes of the test cap connection leaking were: (a) scratching of the seating surfaces due to foreign material; and (b) deformation of the tee and galled threads on the tee due to excessive over-tightening of the cap, the reference leg leak occurred some time between February 22, 1997, and prior to this event similar events had occurred at other plants with no apparent corrective action having been taken at the site as a result, and timely corrective actions as a result of past industry-related events would have prevented the May 3, 1997, even The SEIT concluded that there were no negative issues in a number of areas, including; adequacy of control room staffing, number of concurrent activities and workload on the control room personnel, fitness-for-duty of operations personnel, event classification, recent experience of involved operations personnel in performing unit shutdown activities, FIP effectiveness and timeliness of activation, site management involvement, oversight and conservative decision making, compliance with Technical Specifications during and after the event, plant equipment performance in response to the event, adequacy of I&E procedures involved in work on the LDST level instrumentation, completeness of work packages used for calibration and work on the LDST level instrumentation, simulator training provided to operations of the abnormal procedures used during the event, and accuracy of control room logs in reflecting LDST makeu The AIT reviews of the licensee's event investigation team's processes and results judged the FIP and SEIT to be generally thorough and comprehensive. However, the AIT independent reviews reached some different conclusions in certain area Interviews with the operators on shift at the time of the event, revealed that the operator at the controls, in charge of the overall operations of the unit, was concurrently the dedicated LTOP operator. These concurrent responsibilities had the potential of affecting that individual's performance. Interviews with the operator

determined that he had noted the low LDST level at the beginning of the shift and had intended to add water, but was busy and forgo The AIT reviewed the adequacy of the work packages for calibration and work on the LDST level instrumentation. Even though procedures exist to direct technicians on the proper method to install caps on test connections, these procedures were not referenced or used in the work packages. Licensee metallurgical analysis revealed that inappropriate maintenance practices, such as mismatching parts and over tightening, were a major contributor to the even The licensee documented the issues discussed in this section in PIP 3-097-146 Conclusions The licensee's event investigations using a FIP team and a SEIT was considered to be generally thorough and comprehensive, with some exceptions noted by the AI Exceptions noted were: (1) the licensee's inadequate reviews of past industry events for potential generic problem areas; (2) a lack of implementation of corrective actions to prevent similar occurrences at ONS was a contributor-to the May 3, 1997, event; (3) operator interviews by the licensee event teams could have been more beneficial if they had been conducted individually or individual statements obtained; and (4)

maintenance processes allowed for work packages which did not contain references or requirements to use procedure IV. Plant Support P1 Conduct of Emergency Preparedness Activities P Entry into Notification of Unusual Event Inspection Scope (93800)

The AIT Charter required the AIT to "assess the licensee's performance related to notifications and reporting (i.e., classifying this event, offsite notifications, on-site response and interface with offsite emergency agencies)." The AIT reviewed the licensee's performance related to notifications and reporting (i.e., classifying the event, offsite notifications, on-site response and interface with offsite emergency agencies). Observations and Findings EVENT DECLARATION The Onshift Manager (OSM) declared a Notification of Unusual Event (NOUE) at 3:04 p.m., on May 3, 1997. This discretionary declaration was based on potential degradation in the level of safety of Unit 3 due to the HPI pump problems. Procedure RP/O/B/1000/01, "EMERGENCY CLASSIFICATION," Revision 4 did not require an event declaration for this event. The licensee decided to activate its emergency response facilities to provide technical support to the operating crew on Unit 3 with

their response to the HPI pump problems. The licensee began activation of its emergency response facilities at about 10:30 a.m., on May 3, 1997. The Technical Support Center, Emergency Operations Facility, and Operations Support Center were staffed at 11:10, 11:24, and 11:04 a.m., respectively, and activated following the NOUE at 3:41, 3:55, and 3:41 p.m., respectively. The EOF was de-activated shortly thereafter. The licensee terminated the NOUE at 7:46 p.m., on May 5, 199 EVENT NOTIFICATIONS The licensee did not make a 10 CFR 50.72 notification regarding this event prior to the declaration of the NOUE, nor was such notification clearly required. The licensee's notifications for the NOUE were timely. State and local notifications were accomplished within 15 minutes, and the NRC was notified 43 minutes after event declaration. The licensee provided updated information approximately every 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> during the event. The licensee's Emergency Coordinator approved each notification form prior to its release. The AIT's review indicated the information contained on the notification forms was accurate. At the licensee's Emergency Preparedness critique on May 6, 1997, local offsite officials expressed satisfaction with the licensee's notifications and communications during the event. Although not required for an NOUE, the licensee activated the Event Response Data System during the event to provide detailed plant information to the NRC's Headquarters Operations Cente The licensee notified the Resident Inspector staff shortly after the HPI pump problems on the morning of May 3, 1997, and the Resident Inspectors reported to the sit NRC Headquarters and Region II were aware of the HPI pump problems prior to the NOUE declaration as evidenced by the NRC entering the "monitoring phase" at 1:50 p.m., on May 3, 199 On May 6, 1997, at 6:56 p.m., the licensee submitted a facsimile reporting that a licensee engineering evaluation of the inaccurate LDST level indication determined on May 5, 1997, that the ONS Unit 3 HPI system would not have been able to perform its intended safety function [mitigate a small break loss of coolant accident] during power operating conditions from February 22, 1997, until May 3, 1997. The AIT considered this determination was either immediately reportable under 10 CFR 50.72 (c)(2)(i) or (iii), or four-hour reportable under 10 CFR 50.72(b)(2)(i) or (iii)(D). Conclusions The AIT concluded the licensee's performance related to notifications and reporting (i.e. classifying the event, making offsite notifications, on-site response, and interface with offsite emergency agencies) was good. One exception was noted regarding the timeliness of reporting two days after the event as required by 10 CFR 50.7 V. Event Assessment Al Augmented Inspection Team Event Assessment A Root Cause Evaluation of Degradation of High Pressure Injection Pumps Inspection Scope (93800)

The AIT Charter required the AIT to "Assess the generic aspects of related operations/inspections of all three ONS Units as it relates to: Root Cause of the Unit 3 HPI System degradation, including acts of commission or omission that led to or contributed to the initiation or severity of the event. (Include human factor aspects related to the root cause and response to this event.)" The AIT reviewed all findings and conclusions from the other sections of this report and integrated the items to determine the root causes of this even Observations and Findings The AIT integrated the findings from the other sections of this report and applied an evaluation approach using a failure mode analysis. The following summaries provided supporting information for this approac OPERATIONS FINDINGS/ISSUES Operations area findings included (1) operation during cooldown without appropriate operations crew sensitivity to reactor coolant system and associated systems inventory balancing, (2) plant cooldown procedure weaknesses, (3) adequacy of annunciator response procedure guidance if indications were accurate, (4) abnormal procedure weaknesses for specific event, (5) operator performance during initial knowledge based response to event could be improved, (6) Operations Management Procedure guidance for use of procedures during operation sent a mixed message to operators which may be indicative of toleration of weak procedures, and (7)

plant/operations response after determining that potential pump damage occurred was good, and used appropriate management/technical/emergency preparedness resources for recover MAINTENANCE FINDINGS/ISSUES Maintenance area findings included: (1) calibrations were performed on level instrumentation during operation, which rendered one of two instruments inoperable during that activity; (2) operations logging of calibrations were not accomplished; (3) maintenance practices for work associated with instrumentation fittings were weak; (4) work was allowed to be accomplished by craft without adequate procedure or skills to complete work as required; (5) lessons learned from past industry and plant events not effectively implemented; and (6) configuration control of letdown storage tank level and pressure instrumentation root valves, which were under operational control of the maintenance department, was questionable due to a lack of documentation for initial valve positioning and mislabelling for six of 12 valve ENGINEERING FINDINGS/ISSUES Engineering area findings included: (1) a common mode failure vulnerability for letdown storage tank level instrumentation; (2) deficiencies associated with the letdown storage tank level instrumentation lines including incorrect identification of valves when compared to instrumentation drawings and line sizing discrepancies when comparing as constructed instrumentation piping to design drawings; (3)

licensee lack of understanding of the design basis of a continuous fill line attachment to the instrument reference leg line; (4) unimplemented design changes dating back to 1980, which could have prevented the event; (5) inadequate reviews of past industry events for potential generic problem areas and a lack of implementation of corrective actions to prevent similar -occurrences. Licensee investigation team efforts after the event were generally thorough and comprehensive; however, licensee operator interview techniques could be improve PLANT SUPPORT FINDINGS/ISSUES Plant Support findings included good performance in response to the event using the emergency plan to assemble necessary management, technical and emergency response personnel to support unit recovery after potential pump degradation was identified. However, one event report associated with operating outside the Unit 3 design basis was not accomplished as required by 10 CFR 50.7 *

SUMMARY OF ANALYSIS The AIT used failure mode analysis techniques including barrier analysis and generic problem analysis reviews to evaluate common errors associated with AIT findings and conclusions. Based upon these reviews, the AIT determined there were certain ONS weaknesses that contributed to the root cause. These were: Management ineffectiveness in assuring past industry problems or events were reviewed and appropriate corrective actions implemented, and less than fully adequate root and common cause analysis was identified for the Operations, Maintenance, and Engineering areas. In addition, inadequate performance monitoring and trending was was identified for the Operations areas. The AIT concluded, based on the above analysis, the root cause of the event was inappropriate use of plant and industry operating experience to assure that plant design, maintenance, and operation were focused on reliable operation of the HPI system/components involved in this even Conclusions The AIT determined the root cause of the event was inappropriate use of plant and industry operating experience to assure that plant design, maintenance, and operation were focused on reliable operation of the High Pressure Injection System/Components involved in this even VI. Management Meetings X1 Exit Meeting Summary The inspectors presented the inspection results to members of licensee management at the conclusion of the inspection on May 9, 1997. The licensee acknowledged the findings presente On May 16, 1997, a public meeting was held at ONS to allow the licensee to discuss this event, and their corrective actions prior to restart of Units 2 or Partial List of Persons Contacted Licensee

• E. Burchfield, Regulatory Compliance Manager

• M. Cash, Compliance Manager, McGuire Nuclear Station

• J. Davis, Engineering Manager

• W. Foster, Safety Assurance Manager

• J. Hampton, Vice President, ONS D. Hubbard, Maintenance Superintendent

• C. Little, Electrical Systems/Equipment Manager

• R. Lingle, Operations

• B. Loftis, Electrical Systems/Equipment Supervisor

• B. Peele, Station Manager

• G. Ridgeway, Unit Operations Manager

• T. Saville, Mechanical Systems/Equipment, FIP Team Leader J. Smith, Regulatory Compliance NRC D. Billings, Resident Inspector E. Girard, Reactor Inspector N. Salgado, Resident Inspector

• M. Scott, Senior Resident Inspector

• Attended exit meeting on May 9, 1997

Inspection Procedures Used IP93800

"Augmented Inspection Team Implementing Procedures" dated July 7, 1989

List of Acronyms AIT

-

Augmented Inspection Team ARG

-

Annunciator Response Guidelines BWR -

Boiling Water Reactor BWST -

Borated Water Storage Tank CFR

-

Code of Federal Regulations CR

-

Control Room DPC

-

Duke Power Company EOF

-

Emergency Operations Facility FIP

-

Failure Investigation Process gpm Gallons per Minute HPI

-

High Pressure Injection HPIP

-

HighPressure Injection Pump IN

-

Information Notice LDST -

Letdown Storage Tank LER

-

Licensee Event Report LTOP -

Low Temperature/Over-pressure NOUE -

Notification of Unusual Event NRC

-

Nuclear Regulatory Commission OMP -

Operations Management Procedure ONS

-

Oconee Nuclear Station ONSD -

Oconee Nuclear Site Directive

.

OSC -

Operations Support Center OSM -

Onshift Manager PDR

-

Public Document Room psig

-

Pounds per Square Inch Gage PWR -

Pressurized Water Reactor RC

-

Reactor Coolant RCP

-

Reactor Coolant Pump RCS

-

Reactor Coolant System RO

-

Reactor Operator SEIT

-

Significant Event Investigation Team SPOC -

Single Point of Contact SRO -

Senior Reactor Operator SSF

-

Safe Shutdown Facility STA

-

Shift Technical Advisor TSC

-

Technical Support Center UFSAR

-

Updated Final Safety Analysis Report USQ

-

Unresolved Safety Question WO

-

Work Order

SA'R REG UNITED STATES NUCLEAR REGULATORY COMMISSION REGIONil ATLANTA FEDERAL CENTER 61 FORSYTH STREET, SW, SUITE 23T85 ATLANTA, GEORGIA 30303 May 5, 1997 MEMORANDUM TO:

William E. Holland Team Leader Augmented Inspectio eam FROM:

Luis A. Reyes Regional Adm*

V-s7 rator SUBJECT:

AUGMENTED INS CTION TEAM CHARTER An Augmented Inspection Team (AIT) has been established to inspect and assess the degradation of the Oconee Unit 3 high pressure injection (HPI) system during a plant cooldown on May 3. 1997. The team composition is-as follows:

Team Leader:

' Holland (RII)

G.,Humphrey (R1H)

T.'Cooper (RII)

J. Kaufman (AEOD)

The objectives of the inspection are to: (1)

determine the facts surrounding

.the specific event; (2)

assess licensee response to the event: (3)

assess licensee activity during their event.review and recovery; and (4)

assess generic aspects of operations/inspections that may have broad applicability to the Oconee Unit For the period during which you are leading this inspection and documenting the results, you shall report directly to me. You should control the inspection so that you can conduct an exit with the licensee on May 9, 1997, and a public exit by May 16, 1997. The guidance of Inspection Manual Chapters 0325 and 0610 apply to your inspection and the report. If you have any questions regarding the objectives or the attached charter, contact m Attachment: AIT Charter cc w/att:

H. Thompson, DEDO D. LaBarge, NRR E. Jordan. DEDO R. Wessman, NRR S. Collins, NRR C. Casto, RII D. Ross, AEOD R. Carroll, RII S. Varga, NRR J. Johnson. RII H. Berkow, NRR J. Jaudon, RII G. Tracy, OEDO M. Scott, RII J. Strosnider, NRR

.

A:HHTA 1 of 2

AUGMENTED INSPECTION TEAM CHARTER OCONEE NUCLEAR STATION UNIT 3 HPI SYSTEM DEGRADATION The objectives of the inspection are to: (1)

determine the facts surrounding the specific event: (2)

assess licensee response to the event; (3)

assess licensee activity during their event review and recovery; and (4)

assess generic aspects of operations/inspections that may have broad applicability to the Oconee Units. To accomplish these objectives, the following will be performed:

Develop a sequence of events associated with the degradation of the Oconee Unit 3 HPI system during a plant cooldown on May 3, 199 *

Monitor and review the licensee's activities related to the event investigation (e.g., failure of HPI pumps 3A and 3B, root cause

analysis, precursor event reviews, etc.). Evaluate the effectiveness of the FIP and SEIT teams insofar as they complete their tasks within the time period of your inspectio *

Assess the licensee's activities related to event recovery (i.e., actions to recover plant systems and establish contingencies to restore plant cooldown).

• Assess the licensee's performance related to notifications and reporting (i.e., classifying this event, offsite notifications, on-site response and interface with offsite emergency agencies).

• Assess generic aspects of related operations/inspections of all three Oconee Units as it relates to:

-

Root cause of the Unit 3 HPI system degradation, including acts of commission or omission that led to or contributed to the initiation or severity of the event. (Include human factor aspects related to the root cause and response to this event.)

-

Single failure or common cause vulnerabilities of the letdown storage tank level indication/control syste Availability of procedures addressing contingencies for such problem Damage assessment of the failed HPI pumps and their ability to function after the even *

Document the inspection findings and conclusions in an inspection report within three week Attachment ATTACHMIT A 2 of 2

- -

M

-- c " I 11IAUP. PAESSLAIZER IC tEOW ACLOP SEAL PUP"NZZE24 LP-55

4

5

-~

NOT IA S

TT SNORAAE TAMITON 194l I1EC 1,3C-R2 LETDOWNI NPSAIS STOAG AS-AS

(SI ISA DON

--

-

EL N CI r,11P

20

FILTERP A

C PW V AZ I.~~~~~~~ELI COTIPN SOAIN TP IC VLE TYICl FOR 1NIT.

1 ALL VS VSirECPTA OV0

___

__

13 13

3m.~~~~~~~~~~~

~~

~R

..

U4.. 0.L 1~

LCRC~________

wEIf.mSAI-4-

~ ~ ~

~

~

~

~

~

~

~

~

~

~

~

P mPko IASL

~soWCUIC

~

-ea~a

..

.

IO TAG SEA

________

a..,

142 5-~

.

1O-P TI PUM a

~

T~

FITE 8m III

~i N

--

PO OTS.VlA 4AM

.

T TIODI SS.AI EO-S, 3. 2..t5SPPRIIaEBS IMI.~~~~~~~~~AI W3911 P-EO I

S F-II-I

.21 I

ECO OOMTSS

-~~~~~~~~-~~(

043V4SU-SZ-..21.10TILIIINCIePT9 BO IAIF WATER tS-~~S PC apo-02'i

.t

.~

RATR EIIN iII)

Al TTOAG TEANKT 40E1 SIOI SEAL_________________

___________Cf oICSA

age i or Z Ch-'ecked Control Copy Date/Time OP/3/A/1104/02 ENCLOSURE LETDOWN STORAGE TANK Pressure vs. Indicated Level (Instrument Error Included)

to

30

5 too 10 IA I100.0I I

I 11 g.0 8.0-( IfII I I L

i l 1 7 go 6.

the lef 1 rve 1

'

wr renmD:TM declase Ic..I 115.9.

.

.....

C 4.0 3.9 2.97 I ~~~~eo Ia to t

III0 I

II0 1 a cuse, l0

30

50

70

90 100 LDST Indicated Level (inches)

P QVAf

+---IV0086 IV0085 LETDOWN STORAGE TANK

IV0087 HPILT0033P1 HPILT0033P2 Non-QA r

QIA HPIIVO052 X

HPIVO103 X

TO HPI PUMP SUCTION HEADER OCONEE NUCLEAR STATION - UNITS 1, 2, & 3 NOTE:

LETDOWN STORAGE TANK LEVEL SKETCH FOR INFORMATION ONLY INSTRUMENTATION SKETCH MAY 12, 1997 FILE: LDSTSKCH.VSD REV 0 SHEET 1 OF