IR 05000382/1995023
| ML20149L300 | |
| Person / Time | |
|---|---|
| Site: | Waterford  |
| Issue date: | 02/20/1996 |
| From: | Harrell P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV) |
| To: | |
| Shared Package | |
| ML20149L298 | List: |
| References | |
| 50-382-95-23, NUDOCS 9602260160 | |
| Download: ML20149L300 (24) | |
Text
_
.
.
ENCLOSURE U. S. NUCLEAR REGULATORY COMMISSION
REGION IV
Inspection Report: 50-382/95-23 License: N?F-38 Licensee: Entergy Operations, In P.O. Box B Killona, Louisiana Facility Name: Waterford Steam Electric Station, Unit 3 Inspection At: Taft, Louisiana Inspection Conducted: December 20, 1995 through January 12, 1996 Inspector: T. W. Pruett, Resident Inspector i
Approved: hM P. H. Harrail,jActing Chief, Project Branch D Date Inspection Summar_y .
l Areas Inspected: A special, announced inspection to re/iew the licensee's corrective actions for the existence of voids in the auxiliary component cooling water (ACCW) system and to review the maintenance and testing for Containment Spray (CS) Valves CS-125A and -125 I Results:
,
. The licensee proceduralized a work around for the poor design of the ACCW system by revising procedures and developing a repetitive task for ultrasonic testing (UT) to compensate for the system's susceptibility to void formation (Section 1.3). )
- A number of system engineering performance problems were identified:
(1) condition reports (CR) were closed without verification of .
corrective actions, (2) a UT test frequency was developed without any supporting technical data for the rate of void creation, (3) the frequency for UT did not ensure that design engineering allowable values for void size would not be exceeded, and (4) a statement was made that the ACCW system was operated frequently without basing the determination on an actual review of component run times (Section 1.3).
9602260160 960220 PDR ADOCKOSOOOg82
. j
'l
.
j-2-
- Even though licensee personnel were aware of the ACCW system design problems, these personnel identified why the ACCW system remained operable instead of questioning what actions should be taken to provide assurance that the system could perform its intended design function (Section 1.3). ,
i
- Operations personnel noted that the performance of UT of ACCW piping could be considered a work around, but failed to require that corrective i actions be implemented to eliminate the need for periodic UT of the piping (Section 1.3).
,
i e System engineering and operations were unfamiliar with the intent of the limitations on the size of voids in the ACCW system established by '
design engineering. This lack of familiarity resulted in the ACCW j system not being declared inoperable when voids were identified that l exceeded the established limit. This is an example of inadeqeate '
communications between licensee organizations (Section 1.3).
- The licensee established methods for ensuring that the ACCW ,ystem performed satisfactorily during surveillance testing by implementing preconditioning and work around technique These techniques invalidated the results of the surveillance tests on the system and, as a result, the capability of the system to perform its design basis accident response function was not verified and is an apparent violation of Technical Specification 3.7.3 for Trains A and 8 of the ACCW system i being inoperable (Section 1.4.1). '
I
- The licensee thoroughly understood the mechanisms that caused voids in the ACCW system and had numerous prior opportunities to implement corrective actions that could have prevented recurrence; however, no permanent actions were taken. This is an apparent violation of Criterion XVI of Appendix B to 10 CFR Part 50 (Section 1.4.2).
)
- The licensee's testing program for ACCW Pump Discharge Check l
.
Valves ACC-108A and -1088 did not comply with the requirements of IWV-2200 of Section XI to the ASME Code in that the valves were required to be verified to be leak tight and were not teste This is an apparent violation of Technical Specification 4.0.5 (Section 1.4.3).
- Because degradation of the internals of Valve CS-125A was a contributor of the failure of the valve in September 1993, the lack of documentation !
describing the as-found condition of the valve internals on the work authorization (WA) completed during October 1995, was considered a poor maintenance practice (Section 2.3).
- WA 01135214 had not been reviewed by the originators following the deletion of the acceptance criteria by the system engineer and i identifying in the WA that the mechanical retest was not applicabl ,
These two items are examples of the failure to ensure changes in the l l
.. . . . . . -
.
.
-3-I scope or intent of the WA is reviewed by the originators is an apparent violation of Technical Specification 6.8.1.a (Section 2.3).
Summary of Inspection Findings:
-
New Items-
- Apparent Violation 382/9523-01: Trains A and B of the ACCW system were inoperable (Section 1.4.1)
- Apparent Violation 382/9523-02: Actions were not taken to correct a design deficiency in the ACCW system (Section 1.4.2)
- Apparent Violation 382/9523-03: Failure to properly test the ACCW pump )
discharge check valves (Section 1.4.3)
- Apparent Violation 382/9523-04: Failure to follow a procedure for deletion of requirements from a WA (Section 2.3)
Attachments:
- Attachment 1 - Persons Contacted and Exit Meeting
- Attachment 2 - ACCW System Simplified Diagram i
,
s
_ _ - . . . . . - - . . . _ . . . . - . .. --
- - . . _ _ . . . . . . . . _
.
.
-4-DETAILS 1 REVIEW 0F THE AUXILIARY COMPONENT COOLING WATER (ACCW) SYSTEM (71707, 61726,37551)
1.1 Backaround Information NRC initiated this inspection, originally, to review an inspection followup ite In NRC Inspection Report 50-382/94-05, an issue (Inspection Followup Item 382/9405-02) was identified that involved the lifting of a relief valve on ACCW/CCW Heat Exchanger B on January 28 and 30, 199 The issue was to be reviewed by the inspector to determine the appropriateness of the actions taken to address the unexpected lifting of the relief valv The inspector performed reviews to independently confirm that the licensee had completed the actions identified in the root cause analysis. The analysis specified that three actions should be taken: (1) rerate the system for 125 psig service, (2) replace the spring in the relief valve, and (3) perform UT of the piping to verify that voids were not present in the system. The inspector verified that the licensee had appropriately completed the first two specified actions; however, when the inspector reviewed completion of the third item, numerous problems were identified, as discussed belo .2 S_ystem De:cription The ACCW system is the ultimate heat sink and was designed with the specific function of removing heat from the plant following a loss-of-coolant accident, in order to place the plant in a cold shutdown condition. The system consists of two 100 percent cooling capacity trains, which includes a wet cooling tower basin, a pump, a heat exchanger, and an evaporative, wet-type mechanical draft cooling tower. The component cooling water (CCW) system removes heat from plant components and transfers the heat to the ACCW system via an ACCW/CCW heat exchanger during normal and accident conditions. The ACCW system is operated whenever the heat rejection capacity of the CCW system is exceeded during accident conditions or whenever ambient conditions prevent the CCW system from rejecting all of the normal operating heat load via the dry cooling towers. During the winter months when environmental temperatures are ,
low, the ACCW system is maintained in a standby condition, ready for automatic
. initiation in response to an acciden The system is constructed such that an ACCW pump, located at the minus 32-foot elevation, takes a suction from the wet cooling tower basin and discharges the i water through the ACCW/CCW heat exchanger, located at the plus 28-foot i elevation. After passing through the ACCW/CCW heat exchanger, the water returns to the basin, where it repeats the cycle. See Attachment 2 for a simplified diagram of the syste !
. .. . .
. . - - - - -. - . __ _ . . - -- - .------- _-
.
.
,. -5-
i i
l The approximately 60-foot elevation difference between the basin and the heat exchanger can cause water to drain out of the system into the basin and can result in the creation of voids in the 16-inch diameter inlet and outlet I piping of the. heat exchanger. The rate of void creation is dependent on the i leak tightness of the upper portions of the system piping and components. The l sources of void creation are: (1) air comes out of solution when a vacuum l forms in the ACCW system by column separation, (2) air drawn into the ACCW
!
system through leaking valve packing when a vacuum is developed in the ACCW system by column separation or leakage from the system, and (3) air enters the system during maintenance and is not fully vented after the maintenance is
'
.
complete The rate of void formation is directly dependent on the extent of the system leakage.
!
In addition to the creation of voids due to air leaks into the system, voids l can also be created by column separation in the approximately 60-foot length l of vertical piping from the pump elevation to the heat exchanger elevatio Column separation occurs when water leaks out of the system and there is no source of air to fill the void that is created. As a result, a vacuum is
! created in the void space. At the bottom of this vertical piping, a pump
!
discharge check valve (Valves ACC-108A and -1088) is installed and the function of the check valve is to ensure that water does not drain from the vertical piping back into the basin and create a voi The creation of voids in this system will result in a water hammer event when ;
i
'
the system is started from its normal, standby condition. Water hammer is a pressure rise in a pipe caused by a sudden change in the rate of flow or a stoppage of flow in the pipe. A water hammer event can potentially cause l excessive stresses on the system, which may result in damage to the piping, piping supports, and system components. The magnitude of a water hammer event 1 resulting from a vacuum void is much greater than a void resulting from the
'
presence of air since no air is present to cushion the effects of the generated forces.
,
1.3 Historical Review of Events and Problems with the ACCW System i
I The inspector reviewed the history of problems, as documented by the licensee, with the performance of the ACCW system and identified a number of events, as
,
!
listed below:
- Problem Identification Report 10274 - March 21, 1986 l
- Air-Induced Water Hammer Overpressurization Susceptibility Review -
l November 30, 1993
- Condition Report (CR)94-072 - January 30, 1994
- Root Cause Investigation 94-003 - April 25, 1994
- CR 94-676 - July 14, 1994
,
'
- CR 95-059 - January 13, 1995
- Root Cause Analysis Report for CR 95-455 - December 1, 1995
- * CR 95-1259 - December 4, 1995*
L_ - .- . _
_ _ . .
. . _ . . _ . . _ _ _ . _ _ . . - _ _ _ _ _ ._ _ - . _ _
? l l
i i
. !
I
'
-6-l l
- CR 95-1300 - December 12, 1995* I
,
. CR 95-1329 - December 14, 1995*
i
l These CRs were initiated by the licensee as a result of NRCidentified concerns during review of the licensee's response to 1 the identified ACCW system problems
,
i l
'
The documents listed above that pertain to the problems with the operation of i the ACCW system are discussed in detail in the following section L 1. Problem Identification Report 10274 i This report documented the results of a licensee walkdown inspection, which !
was performed in March 1986, following a water hammer event in the ACCW syste The results of the walkdown identified 11 misaligned,. loose, or bent piping supports and restraints on the discharge side of ACCW Pumps A and The licensee attributed the damaged restraints and supports to a hydraulic j transient caused by column separation in the vertical run of piping between -
the pump discharge at the minus 32-foot elevation and the heat exchanger at the plus 28-foot elevatio The column separation occurred when water drained from the piping, through ACCW Pump Discharge Check Valves ACC-108A and -108B, and into the basin. The licensee took actions to address the damage, which included repairing the damaged supports and restraints, and implementing administrative controls that required shutting the ACCW pump discharge valve prior to starting the pump, opening the high point vent valves, and slowly opening the discharge valve to refill the system by sweeping the voids out of the syste To preclude adverse effects as a result of automatic initiation of the system in response to a design basis accident, the problem identification report stated that operations personnel would provide a routine system surveillance by monitoring a temporary pressure indicator installed at the ACCW pump discharge drain valve in each train. The licensee concluded that monitoring the temporary pressure indicator would provide an indication of the existence of column separation in the piping and would preclude future hydraulic transients until system performance was observed and evaluated and permanent corrective actions implemente The inspector requested that the licensee provide information regarding the type and extent of the permanent corrective actions that had been taken in 1986, as was stated on Problem Identification Report 10274, to address the column separation problem. The licensee stated that no documentation was available that discussed any actions taken in response to the event and as a result, a determination of whether or not actions were implemented could not be made.
I As a result of the review of this event, the inspector concluded that the issuance of Problem Identification Report 10274 constituted a prior opportunity for the licensee to resolve the water hammer problems with the
( a
- -
!
.
-7- l l
ACCW system. However, instead of implementing any sort of effective and permanent corrective actions, the licensee revised plant procedures to 4 instruct operations personnel to manually shut the pump discharge valve prior to starting the pump. These instructions constituted a method of using and ,
'
testing the system without taking actions to permanently correct a known deficient condition (i.e., a work around). In addition, the instructions also provided a method of preconditioning the system prior to the system being tested to verify continued operability. In this case, the pump was started with the discharge valve shut and then the valve slowly opened to ensure that all voids were swept from the system prior to operation of the system at full fl ow. The intent of surveillance testing is to operate the system from the ,
configuration that the system would be in when automatically initiated in l response to an event to ensure that the system could perform its design l l functio The inspector established that the pump discharge valve was left in its l normally open position when the system was placed in its standby lineup, which l reestablished the same conditions that had initially caused the problem that
,
resulted in system damage. The inspector also noted that the licensee did not j take any actions to permanently correct the system design deficienc . Root Cause Investigation (RCI)94-003 and CR 94-072 CR 94-072 was written, in January 1994, to document an occurrence related to the unexpected lifting of the relief valve (Valve ACC-1218) on ACCW/CCW Heat Exchanger B during the performance of testing. _To address this problem, the licensee initiated RCI 94-003, in April 1994, to identify the root causes of the problem and to specify the actions that would be taken to correct the proble The _ licensee completed the root cause analysis and determined that the following items contributed to the inadvertent lifting of the relief valve:
,
l
The 88-psig shutoff head of the ACCW pumps exceeded the 75-psig design rating of the ACCW system and the relief valve setpoint. Design engineering attributed the root cause for the ACCW pump discharge-l pressure being greater than the system design pressure to an inadequate review of system parameters during the original design proces To address this issue, the licensee completed the appropriate analyses and increased the design pressure rating of the ACCW system from 75 to 125 psig. The licensee concluded that the pressure transient that was experienced, as documented on this CR, did not affect ACCW system operability. The inspector reviewed the licensee's actions associated
'.
with this problem and noted that they appeared to be appropriat e The licensee determined that the cadmium-plated, carbon steel spring,
>
used in Relief Valves ACC-121A and -121B, had corroded, caused the spring to weaken, and resulted in a lower setpoint.
t
_- - __
. . .. . _ . . - -. . = - -.
.
-
i-8- 1
!
l To ensure the setpoint of the relief valves did not change as a result I of spring corrosion, the licensee replaced the carbon steel spring with I a stainless steel spring. The inspector reviewed the licensee's actions I associated with this problem and noted that the actions appeared to be l appropriate, i l
- The licensee also identified that the corrective actions specified in l Problem Identification Report 10274 (see Section 1.2.1 of this inspection report), which involved preventing the entrapment of air in the ACCW system, had not been fully implemente The root cause I analysis determined that a review of operating procedures indicated inconsistency in the control of manual ACCW pump starts in that Procedures OP-903-115(116), " Train A(B) Integrated Emergency Diesel l ~ Generator / Engineering Safety Features Test", and OP-903-029, " Safety l Injection Actuation Signal Test", did not provide guidance on shutting
'
the pump discharge valve prior to starting the pump. The licensee revised these operating procedures, which had not been identified in the actions specified in Problem Identification Report 10274, to require the !
pump discharge valve be shut prior to starting an ACCW pum During review of this item, the inspector noted that the licensee took actions to continue addressing the problem of air intrusion into the ACCW system by revising additional procedures to compensate for the susceptibility of the i system to column separation and air intrusion. These actions did not 1 ( eliminate the work around and preconditioning actions that had been established 8 years earlie RCI 94-003 indicated that system engineering would develop a plan to periodically monitor the status of the ACCW piping to ensure that voids of an unacceptable size were not present. Design engineering performed calculations i
that addressed the operability of the ACCW system, based on the size of the i voids present in the system and the physical configuration of each ACCW train.
j As a result of these efforts, the maximum allowed void size was determined to ,
'
be:
- ACCW/CCW Heat Exchanger A - 7.6 inches of arc on the inlet piping and l
25 inches of arc on the outlet piping
. ACCW/CCW Heat Exchanger B - no voids on the inlet piping and 25 inches of arc on the outlet piping.
l The values for the acceptable void size, on the outlet of the heat exchanger, reflect additional conservatism from the design calculations, in that, the i design calculations assumed empty outlet piping. Design engineering stated i that the void assumptions used in these calculations were based on as-found data from 1994 and that calculations were not performed to determine the
.
'
maximum void size on the inlet piping that would result in damage to ACCW/CCW Heat Exchangers A and B.
i i
.
.
-9-Inches of arc was the method established by the licensee for measuring the size of the void in the pipin To determine inches of arc, a measurement is taken from the water / void interface on one side of the pipe, around the outside of the pipe, to the water / void interface on the other side of the pipe. The water / void interface was determined by the use of UT. This measurement then represents the size of the void measured as the length of the pipe diameter ar l System engineering established a repetitive task to perform quarterly UT of l the ACCW piping, in order to determine the size of any voids that might be present, beginning in the fourth quarter of 1994. The once-per-quarter frequency was established such that it would be performed when the routinely
< scheduled ACCW pump surveillance test was performed, instead of being established based on the rate of void formation.
- The inspector reviewed the actions taken by the licensee to implement a j regularly scheduled test for the detection of voids in the system and made the j following observations:
.
'
- The establishment of the repetitive task of the performance of UT .
constituted an additional work around of a system design deficiency and !
'
was done in lieu of addressing the specific system design proble * This occurrence was an additional missed opportunity to resolve the
- ongoing problem with void formation in the ACCW system.
- * This event occurred in January 1994. The root cause analysis was not 3~ completed until April 1994, and UT of the piping, the method implemented to identify voiding in the system, was not instituted until the last quarter of 199 '
1.3.3 Air-Induced Water Hammer Overpressurization Susceptibility Review In September 1993, an event occurred in which Valve CS-125A, containment isolation valve for the containment spray system, failed to open during the performance of a surveillance test. The licensee's investigation identified
- that the failure to open was due to higher than expected pressures in the i system piping as a result of the presence of air in the piping between the pump and the valve. The presence of air caused a pressure increase in the system and prevented the valve from opening due to a high differential pressure across the valve. The details of this issue are provided in NRC Inspection Report 50-382/93-3 To address the generic applicability of this event, the licensee committed to review other safety-related systems, which included the ACCW system, to
'
determine if air entrapment affected operability of the system. The licensee
'
issued a document entitled, " Air-Induced Water Hammer Overpressurization Susceptibility Review," dated November 30, 1993, to document the results of the review of other systems susceptible to an overpressurization event. The
,
-
.
.
-10-results of the licensee's review indicated that portions of the piping in the ACCW system were susceptible to overpressurization during an automatic initiation of the system; however, system pressure protection was provided by a relief valve at the ACCW/CCW heat exchangers and no additional actions were necessar Approximately 6 months after the issuance of the susceptibility review, the licensee issued Waterford Interoffice Correspondence W3C5-94-0124, " Evaluation of Waterford 3 Safety-Related Systems to Minimize Pressure Surges", dated June 1,1994, which was initiated to further review the results of the Air-Induced Water Hammer Overpressurization Susceptibility Review. This memorandum concluded that the susceptibility of the ACCW system to air-induced water hammer overpressurization was very low since an adequate means of i detection existed through surveillance procedures and the system had '
I sufficient relief capacity. The memorandum indicated that: (1) the likelihood of ACCW piping containing entrapped air causing an air-induced water hammer overpressurization problem was low since the piping was swept (i.e., flow through the system removed any voids) during normal operations and quarterly pump operability tests, (2) portions of the piping were susceptible i to column separation and that Electric Power Research Institute's Document 1 NP-6766, for service water cooling systems and auxiliary saltwater systems, provided two design recommendations to mitigate the phenomenon (installation of a vacuum breaker or installation of a keep-fill system), (3) the l installation of a vacuum breaker would compound the air migration problem in i the ACCW system, (4) should corrective actions specified in RCI 94-003 not I prove sufficient to prevent entrapped air, then consideration should be given I to the installation of a keep-fill system, and (5) the leak tightness of the pump discharge check valves (ACC-108A and -108B) was an important design feature in the prevention of void formatio '
Item 4 of the results of the study indicated that the effectiveness of the corrective actions for RCI 94-003 would be the basis for determining whether or not it was appropriate to install a keep-fill system to eliminate void formatio The inspector noted, however, that RCI 94-003 did not provide a time frame for reviewing the effectiveness of the corrective actions and no actions were implemented to periodically review the results of the corrective actions. The inspector concluded that the completion of the overpressurization susceptibility review and the subsequent evaluation, which was documented on Memorandum W3C5-94-0124, constituted two additional opportunities to resolve the ongoing column separation and air intrusion problems with the ACCW syste .3.4 CR 94-676 This CR was issued to document inaccurate system flow readings, during the performance of ACCW Pump A surveillance testing, due to air entrapment in the system. The CR was closed with no actions taken based on the actions that had been taken in response to Problem Identification Report 10274 and CR 94-07 . . . _ _ . . . - - , - . - . . . . - .
l-11-The inspector reviewed this CR and concluded that the licensee had an insufficient basis to close this CR based on corrective actions implemented for CR 94-072 because, had the previous corrective actions been effective, the presence of air in the ACCW system would not have occurred. Additionally, this CR constituted an additional opportunity for the licensee to permanently resolve the ongoing air intrusion problems.
l 1.3.5 CR 95-059 This CR was issued to document the identification of 21 inches of arc in the outlet piping of the Train A ACCW/CCW heat exchanger following filling and venting of the system on January 11, 1995. This CR indicated that the air intrusion resulted from: (1) ACCW Pump Discharge Check Valve ACC-108A leakage, which caused a slight vacuum in the system when ACCW Pump A was secured and allowed entrained air to come out of solution and collect in the
.
system high points, (2) improper venting of entrapped air because l Procedure Op-002-001, " Auxiliary Component Cooling Water," had not been 1 revised to specify the length of time necessary to run the pump to ensure all voids were swept from the system, and (3) a practical method to ensure that ;
all the voids were removed, other than UT of the piping, did not exis l The corrective actions specified on this CR initially involved revising procedures to require a UT following system maintenance and prior to the performance of the ACCW pump quarterly surveillance test. However, operations personnel determined that proceduralizing UT of the ACCW system, prior to performance of surveillance testing, would be considered a work around and would not necessarily prevent a water hammer from occurring should automatic initiation of the ACCW system occur. On July 17, operations personnel directed that the once-per-quarter frequency for UT of the ACCW system be changed to once per month to ensure that the ACCW system would not be susceptible to a water hammer event following idle period The inspector reviewed the actions taken in response to this CR and concluded:
(1) operations personnel continued to proceduralize a work around to
,
compensate for a system design deficiency, and (2) the response to this CR was l not timely in that the corrective actions were not developed during the period l in which the system was most susceptible to air intrusion. This CR was
'
written in January, a time when the system is idle and most susceptible to void formation, and actions were not identified until July, a time when the system was routinely operated and void formations were not a proble In addition, the inspector also noted that this CR represented an additional opportunity to resolve the ongoing column separation and air intrusion problems.
l 1.3.6 Root Cause Analysis for CR 95-455 CR 95-455 was initiated in December 1995, in response to air intrusion in the turbine cooling water system and generic concerns involving air intrusion into other safety-related systems. This CR described two different mechanisms for
air intrusion into the ACCW system, which involved: (1) improper venting of l _ _
_ _ . . _ . _ _
.
.
-12-
!
'
the system following maintenance and (2) column separation in the vertical piping. This CR indicated that the corrective actions from CRs94-072 and 95-059 (issued in January 1994, and January 1995, respectively), once implemented, would adequately prevent the recurrence of air intrusion in the ACCW syste The inspector concluded that the licensee missed an additional opportunity to verify the adequacy of corrective actions specified in CRs95-059 and 94-072.
l In addition, the inspector noted that licensee personnel assumed that the
!
corrective actions from previous CRs had been implemented, without verification that actions had been take . Review of the Corrective Actions Implemented by the Licensee to Address the ACCW System Desian Deficiency
! In December 1995, during review of Inspection Followup Item 382/9405-02, the inspector questioned system engineering personnel to determine what assurances were available to support the previous licensee conclusions that the ACCW system was not susceptible to a water hammer event. System engineering
, personnel stated that they did not believe a water hammer event would occur !
j following automatic initiation of the system because past UT results, which were documented in RCI 94-003, were inconclusive; the rate of void formation
'
between testing intervals varied; and chemistry department sampling required l
frequent operation (at least once per week) of the system and operation of the system would remove any voids before they became excessiv Following these discussions, the inspectors reviewed the validity of these assumption To provide independent verification of the conclusions made by system engineering, the inspector reviewed the UT data provided in RCI 94-003, which had been obtained between January 30 and March 5,1994, and identified five examples of where 8 to 25 inches of arc were identified in the outlet piping, of ACCW/CCW Heat Exchangers A and B within a 1- to 8-day period. The i inspector determined that the void formation in the ACCW/CCW heat exchanger
!
outlet piping could exceed 25 inches of arc within 1 month if the void increased 8 inches every week. An arc of 25 inches would cause the system to be potentially inoperable, as noted by the criteria established by design engineerin The inspector concluded that system engineering had developed quarterly and monthly frequencies for the performance of UT on the piping without any supporting technical basis. The inspector also noted that the data, which were subsequently obtained for the ACCW/CCW heat exchanger outlet piping, invalidated the UT frequencies specified by system engineerin The inspector discussed sampling of the ACCW system with the chemistry superintendent to verify that the system was operated weekly, as stated by system engineering. The superintendent stated that chemistry preferred that the ACCW system be placed in operation periodically for sampling, but did not have any procedural requirements for routine, periodic operation of the ACCW
.. . .. . ._.
,
!.
.
-13-
!
l system. Therefore, periodic sweeping of the voids in the piping may not have l occurre Based on the response from the chemistry superintendent, the inspector again questioned system engineering to determine the frequency of ACCW system operation during normally idle periods. System engineering stated that they
,
did not maintain component run times for the ACCW pumps and did not know when i or how often the system actually operated during normally idle period The l ACCW system engineer stated that he believed the system was frequently
! operated based on plant pager announcements that indicated the system was being placed in operation.
l The inspector reviewed ACCW pump run data for the period of December 1, 1994, I to February 28, 1995. During this period, the inspector identified that ACCW Train A had been started 18 times and there were 5 occurrences of Train A being idle between 7 to 11 days. ACCW Train B had been started 19 times and there were 6 occurrences of Train B being idled between 7 to 13 days. Based on this data, it was apparent that both trains were idle for several long periods of time.
l 1.3.8 Review of the UT Data Obtained for the ACCW System On December 4, 1995, the inspector attempted to perform a baseline analysis of void formation data obtained for the ACCW system but was unable to because of a lack of UT results. The inspector then requested the UT results for all testing performed on the ACCW syste The inspector was notified that
,
additional data were not available in that quarterly testing had not been
! performed during the fourth quarter of 1994 for Train A and during the first quarter of 1995 for Train B, and that monthly testing had not been performed I on either Train A or B during November 1995. Additionally, the inspector determined that the existing UT data for the ACCW system were obtained following maintenance activities and that the quarterly repetitive tasks
,
implemented in response to RCI 94-003 and CR 94-072 and the monthly repetitive l tasks in response to CR 95-059 had never been performe In response to the inspector's identification of the failure to perform UT on l
the piping to detect the presence of voids, system engineering initiated
!
CR 95-1259 and commenced a review of tasks assigned to plant engineering to determine why the testing had not been completed. The licensee stated that testing had not been performed because the task was assigned to plart engineering instead of a maintenance group and because the system engineer did not realize that he was responsible for scheduling the tas Because of the inspector's concerns that the rate of void formation had not i been determined and because there was no technical basis for establishing a UT repetitive task frequency of monthly, system engineering and operations personnel determined it was necessary to perform a baseline study on the rate
,
of void formatio __
.
-14-On December 11, 1995, the licensee initiated UT of the ACCW system piping on a daily frequency and obtained the following data:
TIME TRAIN OUTLET ARC INLET ARC 4:30 A EMPTY * FULL B FULL 6 in.*
6:00 A 8 i FULL B FULL FULL
- This amount of arc exceeded the allowable amount specified by design engineering for system operability When the data were obtained and it was identified that an excessive amount of arc existed in the piping, the licensee started Trains A and B to remove the air voids and initiated CR 95-1300. The inspector reviewed the data and noted that the ACCW system was last operated on December 4 and that, within I week, the void formation had exceeded the design engineering maximum allowable values for continued system operability of 25 inches of arc on ACCW/CCW Heat Exchanger A outlet piping and no voids on ACCW/CCW Heat Exchanger B inlet pipin On December 12, the licensee performed additional UT and obtained the following results:
TIME TRAIN OUTLET ARC INLET ARC 11:00 A 28 in.* FULL 2:35 A 15 i FULL 3:30 A 15 i FULL 4:30 A 15.25 i FULL 5:00 B FULL FULL 5:30 A 15.75 i FULL
- 6
- 30 A 16 i FULL
- This amount of arc exceeded the allowable amount specified by design engineering for system operability The inspector questioned the licensee to determine whether the frequency for
- UT of the piping would be increased since the design engineering maximum allowable value for the outlet piping on the Train A ACCW/CCW heat exchanger had been exceeded within 1 day. The licensee stated that the testing frequency would be increased until there was an assurance that the rate of void formation was know __
. . . - - . .
l'
l
.
-15-l On December 13, the licensee obtained the following UT results:
l TIME TRAIN OUTLET ARC INLET ARC 9:00 A 23.50 i FULL 11:00 A 24.50 i FULL 1:00 A 26.00 in.* FULL 4:30 A 5.50 i FULL 10:00 FULL 4.5 in.*
10:45 B FULL FULL
! * This amount of arc exceeded the allowable amount specified by design engineering for system operability
,
The licensee initiated an operability evaluation since the 4.5 inches of arc exceeded the design engineering maximum allowable value of no voids for
'
l ACCW/CCW Heat Exchanger B inlet piping and the 26 inches of arc exceeded the !
design engineering maximum allowable value of 25 inches of arc for ACCW/CCW 4 l Heat Exchanger A outlet piping. The inspector questioned the licensee to j determine why an operability evaluation had not been initiated on December 11, in response to the 6 inches of arc identified in the ACCW/CCW Heat Exchanger B
,
inlet piping and complete voiding in the ACCW/CCW Heat Exchanger A outlet
- piping. System engineering and operations personnel stated that they did not
'
believe the maximum allowable values for voids specified by design engineering were operability limitations; therefore, the system was not declared inoperable. Instead, system engineering and operations personnel stated that they believed that the values were limitations on when sweeping and venting of l the system should be performe System engineering and operations personnel l l
also stated that an elevated awareness and concern, as a result of the l questions being asked by the inspector, over ACCW system voiding and l operability prompted the December 13 operability evaluation.
l l The inspector noted that the void size criteria established by design
.
engineering for allowable inches of arc were in response to determining the j maximum size void that could exist in each train without damaging the system,
'
if an automatic start of the ACCW pump occurred and the pump discharge valve was open. Design engineering stated that the intent of the maximum allowable values were not adequately documented in RCI 94-003 and that this inadequacy may have contributed to the delay in initiating an operability evaluation for previous out-of-tolerance measurements. The inspector concluded that a lack of familiarity with the intent of the limitations on void size in the ACCW system, by system engineering and operations personnel, was an example of poor implementation of corrective actions.
I
!
i
_ _ _ __ _. __ . _ _ _ ___ _ ._
.
.
-16- !
!
l On December 14, the licensee obtained additional UT results:
l . TIME TRAIN OUTLET ARC INLET ARC 5:00 .25 in.
'
5:40 A B FULL FULL FULL
)
Following this testing, system engineering recommended that operations run the ACCW system continuously, with Temperature Control Valves ACC-126A and -1268 I in the closed position, and daily venting from system high point vents be performed. Maintaining the valves closed minimizes flow through the CCW heat L exchanger and maximizes flow through the pump recirculation line. Operation
- of the system in this manner was expected to last until a permanent solution for the ongoing design problem with void formation was implemented. System engineering stated that this configuration would allow the ACCW system to ,
remain pressurized and free of voids and also prevent overcooling of the CCW !
system by maintaining Valves ACC-126A and -1268 shut. The inspector reviewed the Final Safety Analysis Report description for the operation of the ACCW system and compared this information to engineering's recommendation for temporary full-time operation of the system. The inspector concluded that
'
there were not any apparent detrimental effects on system performance if the ACCW pumps were continuously operated at or near shutoff head and that continuous operation of the system should prevent void formation ir, the ACCW pipin . Review of Desian Engineerina Calculations l On December 4, 1995, in response to CR 95-1300, design engineering initiated a series of calculations to determine the effect of the size of voids on the i inlet piping of Heat Exchanger B. Based on the results of these calculations, i the licensee determined that the dynamic loading would be within design allowables if the void size on the Heat Exchanger B inlet piping was less than or equal to 4 inches of arc. Although calculations were not performed, design engineering assumed that no damage would result from a 4.5-inch arc (the results obtained on December 13) because of the inherent conservatism in the calculation and because the actual yield strength of the material was probably greater than the ASME code yield strength assumed in the calculation.
'
Following completion of the December 13 operability evaluation, design engineering personnel stated that the operability limit for air in the ACCW system was:
Heat Exchanger A - 7.6 inches of arc in the inlet piping and empty in the outlet piping l
Heat Exchanger B - 4.0 inches of arc in the inlet piping and empty in i
the outlet pipin . .. _
3 _ _
- f
.
-17-In addition, design engineering stated that any voids larger than the limits shown above, should be removed in accordance with operating procedure .3.10 Inservice Testino of the ACCW Pump Check Valves The inspector reviewed Section 3.2.3, " Auxiliary Component Cooling Water", of Design Basis Document (DBD) W3-DBD-024, " Inservice Testing Design Basis Document", dated March 1994, and determined that the safety position of ACCW Pump Discharge Check Valves ACC-108A and -108B was active open and active closed. DBD W3-DBD-024 also indicated that the closed safety function of the check valves prevented water hammer and preserved structural integrity of the system. The inspector noted that the licensee's inservice testing program only required that Valves ACC-108A and -1088 be tested to ensure full flow in the open direction and did not specify that testing be performed to verify that the valve was leak tight in the closed position. The inspector concluded that automatic initiation of the ACCW system, combined with undetected leakage
,
'
past Valves ACC-108A and -108B, could have affected the operability of the system. The inspector also noted that the testing program for Valves ACC-108A and -1088 was not commensurate with the importance of its active closed safety positio '
The licensee performed operability reviews of any issues that were identified during the creation of a DBD, and the reviews were included in the DBD as Appendix C. The inspector reviewed the licensee's operability evaluation for not testing the check valves in the reverse direction to ensure that the valves were leak tight. The licensee's evaluation stated that the ACCW system continued to be operable because the ACCW piping was monitored for voids, the system was refilled if any significant voids were identified, and CR 94-072 and RCI 94-003 had been initiated to track voiding in the ACCW syste !
The inspector concluded that past operability of the ACCW system was I indeterminate because operability of the system was directly linked to the I performance of UT of ACCW piping and the licensee had not performed routine UT of the syste i 1.4 Conclusions Based on the reviews the inspector performed of this problem, the following conclusions were identified: I 1.4.1 Operabilit_y of the ACCW System The licensee established that the presence of voids in the ACCW system affected operability of the system in that physical damage could occur if the system was initiated from the standby condition. To ensure continued system operability, the licensee proposed implementation of a program to perform routine UT on the piping to detect the presence of voids; however, the licensee did not perform testing that was adequat When the licensee did initiate UT of the piping, it was identified that the size of the voids in the system exceeded the criteria specified by
_ . ... . . . . - - , _ - .- - ..
.
> .
-18-the licensee's design engineering personnel. In addition, the licensee established that a frequency of quarterly was appropriate; however, the licensee did not initially perform any testing to determine the rate of void formation. As was established in subsequent testing, the rate of void formation indicated that the quarterly frequency was too long in that the size of the voids exceeded the established criteria in approximately 1 da Procedure changes were made to precondition the system before I performance of surveillance testing such that the licensee established system conditions that would not exist if the ACCW system was required to respond to an actual event. For this reason, it is not apparent that i i
the licensee performed adequate surveillance testing of the ACCW system to verify that it could perform its design basis functio Based on the presence of voids in the system and the potential affect on system operability, it is not apparent that either train of the ACCW l system was operable at all times. This is an apparent violation of i Technical Specification 3.7.3, which requires the ACCW system to be !
operable while operating in Modes 1 through 4 (382/9523-01). !
1.4.2 Actions Taken to Correct the ACCW S_ystem Design Deficiencies l
Since 1986, the licensee identified, on numerous occasions, that the ACCW system contained a design deficiency in that voids formed in the system and the presence of voids affected the ability of the system to perform its design basis function. On each of these opportunities, the licensee did not take actions to address and correct the system design deficiency, but instead, implemented methods for continuing to use the system in its degraded condition by establishing work arounds and preconditioning of the syste The failure to take actions to correct a significant condition adverse to the continued operability of the ACCW system and to prevent recurrence of the condition is an apparent violation of Criterion XVI of Appendix B to 10 CFR Part 50 (382/9523-02).
1.4.3 Testing of the ACCW Pump Discharge Check Valves During review of this issue, the inspector noted that the licensee took credit for voids not forming in the ACCW system piping, during the closecut of CRs, because the pump discharge check valves (ACC-108A and -108B) prevented any water from leaking from the system. The inspector established that the licensee was not performing testing of Valves ACC-108A and -108B in the closed direction to verify that the valves were not leaking, as was assumed by the license Technical Specification 4.0.5 requires that inservice testing of valves be performed in accordance with the requirements of IWV-2200 of Section XI to the ASME Code, which requires testing of valves that
._ _ _ __ _ . _ _ _ ..
.
I l.
I-19-l l perform a safety function. The failure to perform testing of i Valves ACC-108A and -1088, as required by Technical Specification 4. is an apparent violation (382/9523-03).
1.4.4 Additional Conclusions l
l The inspector also noted the conclusions discussed below during review of this l
.
issue:
- Licensee personnel identified the reasons why the ACCW system remained operable, even with the identified system design deficiency, instead of l questioning what potential adverse affects the ongoing problems had on i system operability and what actions should be taken to permanently correct the system design inadequacies.
t 1
'
l * CRs were closed by engineering personnel based on actions that were i specified on CRs that had been previously issued. Engineering personnel
!
took credit for these actions; however, they failed to verify that the j actions specified were actually implemented. In a number of cases, the l actions specified by other CRs had not been implemented and, as a result, CRs were closed without addressing the actual ongoing problem * The communications by system engineering and operations personnel with the design engineering organization was not adequate relative to this issue. The design engineering organization established operability
,
criteria for the size of the voids in the ACCW system that would cause the system to be considered inoperable. Voids were identified during UT activities that exceeded the criteria; however, the ACCW system was not declared inoperable because the system engineering and operations personnel were not aware of the significance of the criteri CONTAINMENT SPRAY (CS) HEADER ISOLATION VALVES CS-125A AND -125B (71707, 62703, 61726, 37551)
2.1 Background Information On November 28, 1995, management representatives from NRC Headquarters and Region IV performed a tour of the facilit During this tour, these individuals noted that a Mechanical Retest Required tag was installed on
'
Valve CS-125 The NRC management representatives questioned why such a tag would be installed on the valve since they were aware that the plant had recently started up from a refueling outage, Valve CS-125B was a containment isolation valve, and the licensee had previously experienced operability
.
problems, as discussed above. The representatives requested that the resident i inspector perform a followup review to: (1) determine what type of mechanical
! retest was required, (2) establish why the test had not been completed prior
- to plant startup from the refueling outage, and (3) determine whether the retest affected either the valve or system operabilit .
.
-20-2.2 System Description Valves CS-125A and -1258 are air-operated, 10-inch gate valves with an open safety function to provide flow to the containment spray headers to reduce l containment pressure following a loss-of-coolant accident. Valves CS-125A and-1258 must open within 10 seconds, upon receipt of a containment spray actuation signal, to ensure that containment spray is initiated in a timely manne It is also assumed that the valves will be able to open at a maximum 300 psid across the valve On September 13, 1993, during the performance of surveillance testing,
! excessive differential pressure across Valve CS-125A, as a result of air entrapped in the system between the pump and the valve, prevented
,
Valve CS-125A from opening. The licensee completed an engineering evaluation l of the event and determined that: (1) a 331 to 344 psig peak pressure existed upstream of Valves CS-125A and -1258 following a CS pump start, (2) CS Pumps A
,
and B acceleration time to reach 300 psig was 1.12 to 2.09 seconds, (3) the
! time required for the initiation of flow through Valves CS-125A and -1528 was 1.44 to 1.70 seconds, and (4) a degraded Valve CS-125A failed to open with a l
differential pressure of 225 to 245 psi l In response to the failure of Valve CS-125A to operate as designed, the i
! licensee: (1) installed additional venting capacity for the valve air
'
operator to reduce the stroke time of the valve, (2) installed additional vent valves on system piping to remove entrapped air, (3) replaced the valve internals, and (4) established an 18-month repetitive task to visually inspect the valve internals. The licensee also installed a time delay relay in the
- starting circuit for the CS pump to ensure that Valves CS-125A and -125B would begin to open prior to CS Pumps A and B starting, thereby eliminating the differential pressure across the valv The NRC review of the failure of Valve CS-125A to function as designed is ;
, described in NRC Inspection Report 50-382/93-33. The corrective actions taken '
by the licensee are discussed in Licensee Event Report 93-004.
'
l 2.3 Review of the Work Authorization (WA) for Valve CS-125B The inspector reviewed the maintenance performed on Valve CS-125B during the
'
outage, which was completed in accordance with WA 01135214, to determine why a mechanical retest was required. The inspector noted that the maintenance performed on the valve involved an 18-month disassembly and inspection of the condition of the valve internals and seat o-ring.
l After completion of the inspection and reassembly of Valve CS-125B, Item 1 on j the Postmaintenance Test Sheet contained in WA 01135214, stated, in part, that i
postmaintenance testing be performed in accordance with Section 8.3 of Procedure MM-003-043, " Containment Spray Isolation Valve Inspection and Testing." This procedure required, in part, that the mechanic verify that the valve stroked freely through its full length of trave Based on discussions with the maintenance supervisor and the lead mechanic, the inspector i
l l
.
t
! .
-21-i determined that maintenance personnel manually stroked the valve prior to
- reinstallation of the limit switches and air-supply solenoid valves to ensure l that Valve CS-125B operated freely. The lead mechanic stated that the valve operated freely; however, he did not sign Item 1 on the Postmaintenance Test ,
Sheet because an observation of the valve operation needed to be performed l following complete reassembly (i.e., installation of the limit switches and i the solenoid valves). Because the valve had not been stroked using air pressure to verify full length travel, a Maintenance Retest Required tag was ,
installed on the valve so maintenance personnel would be aware that the test l
needed to be performed at a later dat )
l During review of WA 01135214, the inspector noted that the requirement to perform the mechanical retest was marked NA (not applicable) on the Postmaintenance Test Sheet by the maintenance planner during a review of the WA on November 29, 1995, which was about 3 weeks after the refueling outage ended. The planner annotated on the WA that the basis for marking the mechanical retest NA was that a valve stroke time test had been performed by ,
the operations department in accordance with Procedure OP-903-121, " Safety '
Systems Quarterly IST [ Inservice Testing] Valve Tests." This test verified that the valve opened within a specified time; however, it was not apparent that a stroke time test would identify potentially loose parts on a valve since no visual observation was performed by a mechani During further followup, the inspector reviewed Procedure OP-903-121, Attachment 10.2, " Containment Spray," performed on October 30, 1995, and noted that the procedure stated that an auxiliary operator was present at the location of Valve CS-125B during testing and that any unusual operation of the valve would likely have been detected by the operator. The inspector concluded that the specific intent of the mechanical retest had not been met; however, in this specific case, the presence of tM operator when the valve was stroked was sufficient justification to state that the mechanical retest had been complete Procedure UNT-005-015, " Work Authorization Preparation and Implementation," '
Section 5.6, " Work Performance," stated, in part, that the WA shall be reviewed by the original reviewers affected by a change in scope or intent. A change in the postmaintenance test requirements of a WA was considered a change in the scope or intent. An intent or scope change of a WA without review by the original reviewers is an example of a failure to follow procedure and is an apparent violation of TS 6.8.1.a. (382/9523-04).
Item 2 on the Postmaintenance Test Sheet contained in WA 01135214 stated, in part, that operability testing be performed in accordance with Procedure MM-003-043, Section 8.5, " Containment Spray Isolation Valve CS-1258
! Operability Test Against Design Differential Pressure of 300 PSIG." The j inspector noted that this test requirement was marked NA. A written entry on l the Postmaintenance Test Sheet stated, "During a containment spray actuation l signal the valve will open before pump, so there will be no significant differential pressure. Differential pressure test is not needed for valve l operability." This entry was made by the system engineer on October 27, 1995,
.
.
-22-af ter Valve CS-125B was reassemble The basis for this statement was that a 2-second time delay relay had been installed in the CS pump starting circuit and this relay allowed the valve to open before the pump started, which would eliminate any differential pressure across the relay. Although the engineer took credit for this relay, he did not verify that the relay had been routinely tested to verify it could perform its intended function. Reviews by the inspector established that the relay was not routinely tested. This is another example of an engineer making an assumption regarding the performance of a system without verifying the assumption was vali The inspector noted that WA 01135214 had not been reviewed by the original reviewers following the deletion of the acceptance criteria by the system engineer and is another example of an apparent violation of TS 6.8. (382/9523-04).
The inspector questioned how Valves CS-125A and -1258 were declared operable without the completion of the 300 psid operability test or a test of the time delay relay. On January 10 and 12, 1996, the licensee tested Valves CS-125A and -125B to determine whether the valves would open against a differential pressure of approximately 300 psid. The test was a static pressure test, with the pressure upstream of the valve being provided by a hydrostatic test pump instead of a CS pump. The test results indicated that the valves would open under the simulated conditions, which was an indication that the valves were operabl During review of WA 01135214, the inspector also noted that the mechanics replaced the seat, cleaned the internals, and reassembled the valve. However, the mechanics did not document on the WA, the extent of seat degradation, the amount of deposits on the valve internals, or other information regarding the as-found condition of the valve. Because degradation of Valve CS-125A '
contributed to a September 13, 1993, failure of the valve, the inspector l concluded that a lack of documentation describing the as-found conditions of the valve internals was a poor maintenance practice since, without the information, the licensee could not trend the degradation of the valve. The mechanical maintenance supervisor stated that this observation would be j reviewed and improvements made in documenting the as-found condition of I components disassembled for inspection, as appropriat I i
. - - - - . . . --- . .. ..
.
O ATTACHMENT 1 1 PERSONS CONTACTED Licensee Personnel i
R. Allen, Manager, Operational and Engineering Experience R. Azzarello, Director, Design Engineering R. Barkhurst, Vice-President, Operations
, R. Burski, Director, Nuclear Safety G. Davie, Quality Assurance Manager M. Ferri, Director, Plant Modification and Construction G. Fey, Supervisor, Quality Assurance
,
T. Gaudet, Supervisor, Licensing ;
J. Hoffpauir, Maintenance Superintendent '
A. Holder, Senior Engineer, Fire Protection J. Holman, Safety Analysis J. Houghtaling, Technical Services Manager J. Hologa, Manager, Mechanical and Civil Engineering D. Keuter, General Manager, Plant Operations J. Laque, Supervisor, System Engineering J. Ledet, Security Superintendent D. Litoff, Licensing A. Lockhart, Quality Assurance Manager D. Marpe, Mechanical Maintenance Supervisor D. Matheny, Operations Superintendent J. O'Hearn, Manager, Training W. Pendergras, Shift Supervisor, licensing '
R. Peters, Electrical Maintenance Supervisor R. Pittman, I&C Maintenance Supervisor B. Proctor, Superintendent System Engineering Mechanical D. Shipman, Planning and Scheduling Manager R. Starkey, Manager, Operations and Maintenance F. Titus, Vice-President, Engineering D. Vinci, Licensing Manager J. Yelverton, Executive Vice-President and Chief Operating Officer NRC Personnel S. Collins, Deputy Regional Administrator, Region IV P. Harrell, Acting Chief, Branch D, Division of Reactor Projects, Region IV The personnel listed above attended the exit meeting. In addition to these personnel, the inspectors contacted other personnel during this inspection perio EXIT MEETING An exit meeting was conducted on January 11, 199 During this meeting, the inspectors reviewed the scope and findings of the report. The licensee did not express a position on the inspection findings documented in this repor The licensee did not identify as proprietary any information provided to, or reviewed by, the inspector _ 4
- - _ - _ _ _ _ - - - - - - _ _ - _ _ __ _ _ _ _ - _ _ _ _ _ _ _ - . _ _ . ____ ______-____--___ -__ .
..
o O
ATTACHMENT 2
.
.
.
.
- n +# 9'
...n,. u, w s.-...
. . . n .or
,
l Essentialchht A f a * - - EL * W 8* .
= = u .se !
. 8 EL +34' 9". . -
,, #T
'
ACC 8404 ACC.13th f 1P ACC43 C tI dL Jb
.w n.,,.... ,
" g . .
g g A 4344 ACCat37A 4 if a
k ACC tilA B
.
CCW LesBaebaseer A - - - K .2s- CC' 0" Ccw am 1. mc .-
qr -N ACC419A Jk
..
n .ir . . .
-
g t**&
. c a .. < -....r
^**"'^
w cw..,2.....a N - .M-Accai Act.i..A
. ":' r .
ACC 4 t4A ACC IGIA Te EfW ACCW hssp A Simplified Elevation View of the CCW/ACCW Systems l
__-___-___