IR 05000213/1985020

Insp Rept 50-213/85-20 on 850918-23.Major Areas Inspected: Circumstances & Corrective Actions Re Auxiliary Feedwater Actuation Component Failure.Mgt Meeting Scheduled to Discuss Corrective Actions

ML20138L078
Person / Time
Site:	Haddam Neck File:Connecticut Yankee Atomic Power Co icon.png
Issue date:	10/24/1985
From:	Mccabe E, Swetland P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I)
To:
Shared Package
ML20138L070	List: IR 05000213/1985020 ML20138L065
References
50-213-85-20, NUDOCS 8510310167
Download: ML20138L078 (8)

Text

.

-

.

U.S. NUCLEAR REGULATORY COMMISSION REGION I DCS N Report N /85-20 Docket N License N DPR-61 Licensee: Connecticut Yankee Atomic Power Company P. O. Box 270 Hartford, CT 06101 Facility Name: Haddam Neck Plant, Haddam, Connecticut Inspection at: Haddam, Connecticut Inspection conducted: September 18-23, 1985 Inspector: @* b%. b P. D. Swetland, Senior Resident Inspector sol 24lPT Date Approved by: $bN E. C. McCabe, Chief, Reactor Projects Section 38 to It4ISS Date Inspection Summary: Special onsite inspection (27 hours3.125e-4 days <br />0.0075 hours <br />4.464286e-5 weeks <br />1.02735e-5 months <br />) to review the circum-stances and corrective actions regarding an auxiliary feedwater (AFW) actuation component failure, incorrect AFW actuation circuit wiring and apparent inade-quate post-modification testing,'and ' potential AFW design basis inadequacies which were identified in conjunction with AFW system testing on September 10, 198 Results: There were three unresolved items identified in this report. The first requires licensee effort to identify the cause and corrective action (s)

for the'SOV failures (Detail 4.1). The second requires NRR review of the licensee's submittal on AFW design basis (Detail 4.2). The third issue involves the SOV'miswiring and apparent ' inadequate post-modification testing (Detail 4.3). A management meeting Was scheduled to discuss ths licensee's

.

actions to resolve the concern G25 PDR ADOCK 05000213 G PDR

-

.

DETAILS

- Summary On September 10, 1985, the licensee tested six auxiliary feedwater (AFW)

system solenoid actuated valves (S0Vs) which are required to function to automatically initiate AFW. This was the first of a series of planned periodic tests conducted because no cause was identified for a previous failure of two of these SOVs in November 1984. During the September 1985

' test, another 50V failure was experienced. Also, unexpected equipment actuations during the test led the licensee to identify a miswiring of power supplies to four out of six'50Vs. During review of these concerns, the inspector was further informed by the licensee that the design basis of the AFW system may now require flow to all four steam generators to remove decay heat. If this is the case, the system is not single failure proof. Areas of concern identified during this inspection and their

_present status are listed below and detailed in subsequent paragraph Repeated similar failures of AFW SOVs without a definitive cause identified raises the question of susceptibility of automatic AFW actuation to common mede failure. After November 1984, no other S0V failures were experienced during system operation; however, one subsequently failed the first test in September 1985. The licensee initiated a progressive frequency test program to establish and maintain reliable SOV operation. Evaluation of possible failure mechanisms and final corrective actions is ongoin N parent errors / omissions in modeling reactor coolant and AFW system performance for the current Loss of Feedwater transient analysis have been identified. Incorporation of the actual system performance will result in less conservative design margins for this analysis. The licensee documented the AFW design basis problems in a letter to NRC Licensing dated September 20, 1985. Based on the availability of alternate paths to supply feedwater to the SGs and time for operators to correct problems and/or utilize the alternate paths, the licensee concluded that continued plant operation was justified pending NRC Licensing evaluation of the design analysis deficiencie Inadequate installation controls and testing during the implementation of the automatic AFW actuation modifications resulted in miswiring the power supplies for four of six AFW SUVs. Since all power supplies to the S0Vs are isolated during any automatic system

actuation, the miswiring did not affect normal system cperatio However, such wiring errors affect the confidence in system as-built configuration information, part.icularly as it applies to prompt and accurate problem diagnosis for plant maintenance or emergency repair activities, as well as for subsequent system modification The ,

licensee is evaluating appropriate corrective actions for the AFW l miswiring problem I

-

-

.

2.0 Background In response to the TMI Action Plan, the licensee installed an automatic AFW actuation syste This system senses steam generator (SG) levels in all four SGs and automatically starts the two turbine driven AFW pumps and opens four main feedwater bypass control valves to direct feedwater to each SG upon low-lcw level (45% on normal wide range indication) in any

'

two of the four SGs. The system consists of separate and redundant (A & B trains)-sensors, logic matrices, and lock-out actuation relays. AFW system operation is controlled by six air-operated valves which control steam to the "A" and "B" turbine-driven AFW pumps and AFW flow to each S These six valves fail to their safety function position on loss of ai Automatic AFW actuation is accomplished when either or both actuation circuit lock-out relays open the power supply to the solenoid-operated air control valves (SOVs) for each of the six air-operated AFW control valve The SOVs open on loss of power to vent control air from their respective control valves, opening them to initiate AFW flow. The S0Vs have vital power supplies and this part of the system " fails safe" on loss of vital AC control power. This design was completed and upgraded to safety grade over the period 1980-1983. NRC reviewed the design as documented in a Safety Evaluation Report dated October 5, 198 .0 Event Description The licensee initiated a remedial test program for the AFW SOVs because two of the six SOVs had failed during Technical Specification required refueling interval system actuation tests performed in November 1984. The S0Vs stuck closed during the initial system actuation test but, until September 1985, functioned normally during subsequent operations. The li6ensee found no apparent cause for the failures. Based on licensee discussions with the SOV manufacturer, the licensee committed to performing remedial testing of the SOVs during the current operating cycle. These tests exercise the S0Vs to provide assurance of system operability and/or better component failure data. All six AFW SOVs functioned properly during a required AFW system actuation following a plant shutdown in May 1985. The licensee developed a procedure for online testing of the SOVs in September 1985. On September 10, 1985, the licensee conducted the first remedial testing of the six AFW S0Vs in accordance with this procedure (9.2-45, Functional Verification of Automatic Initiation of AFW SOVs). The test was interrupted when the "B" turbine driven AFW pump started as a result of pulling the "A" pump actuation 50V fuse. (If either the "A" channel or the "B" channel had been actuated instead, both pumps would have started.) The A pump remained idle. Licensee investigation revealed that four of the six SOVs were miswired, with respect to current plant drawings, such that the A and B pump SOVs were cross-wired, and the Nos. 3 and 4 steam generator feedwater bypass control valve SOVs were also cross wire In addition, the SOV for the #1 feedwater bypass valve stuck closed. This was one of the two SOVs which had failed previously in November 198 Similar to that earlier test, the SOV functioned properly during followup tests, and no specific cause of failure was identifie The licensee initiated a progressive frequency surveillance program starting with daily

-

.

SOV testing to establish an appropriate test frequency to assure reliable system operation. The licensee has currently reached a weekly test frequency with no additional SOV failures experience .0 Findings:

4.1 AFW S0V Failures During three actuations of the AFW SOVs since November 1,~ 1984, SOV failures have been experienced on two occasions. No cause of failure '

has been identified, and a potential for common mode failure exist .No failures have been experienced at the current weekly test frequency. The inspector observed the tests on September 11 and 16, 1985. These tests were conducted in accordance with the approved procedure and no discrepancies were identified. Licensee evaluation of the failure mechanism and appropriate corrective action continue The licensee plans to report this occurrence via the Licensee Event Reporting System. The inspector determined that an alternate means (manual remote control of the air operated valves) exists and ample time before steam generator generator dryout (about 30 minutes) is available for operator action to mitigate the consequences of any AFW SOV failure. The present level of success of weekly SOV testing and the availability of alternate means for actuation provides two measures of assurance of AFW operation. Licensee determination of the cause and corrective actions for the SOV failures has not been achieved yet. This item remains unresolved pending NRC review of the licen-see's event report which is to be submitted. (UNR 50-213/85-20-01).

4.2 AFW System Design Basis During licensee and NRC review of this event, questions were raised regarding the current design basis for the AFW system. First, the licensee has indicated that the current design analysis for Loss of Feedwater included in the Facility Design and Safety Analysis report and updated (for Loss of Feedwater) in a licensee submittal to NRC dated May 19, 1980, did not properly model reactor coolant pump (RCP)

-

operation or current AFW flo Licensee internal correspondence and an August 28, 1985 submittal to NRC Licensing stated that new analyses which correctly model RCP operation and actual AFW flow no longer support a conclusion that AFW flow to only two SGs is sufficient to remove decay heat. The licensee concluded that flow to all four SGs may be required, thereby heightening the significance of the single SOV failure with regard to automatic initiation of AF The licensee later determined that flow to all SGs is not necessary to meet NRC acceptance criteria for this transient. The licensee

'

also concluded that, since current analyses show satisfactory results if AFW is initiated automatically or manually after 10 minutes, and since manual AFW operation was not affected by the S0V failure or mis-wiring, the design basis for AFW was maintained and continued plant operation was justified based on the manual actuation

__. _ .__ _ _ _

_ __ _ --

. . - . - _ - . _

-_

-

.

capability. The inspector verified the availability of the manual AFW actuation capability and that another direct feed path for AFW flow (operable from'the Control Room) to the SGs exists should a control valve fai However, NRC acceptance of the manual AFW actuation capability was not determine The potential need to feed all four SGs also raised a second. question regarding the design basis of the automatic AFW initiation system. The NRC Safety Evaluation Report (October 5, 1982) for the AFW initiation system stated that no credible single failure could prevent fulfillment of the automatic AFW initiation design function. If AFW flow was needed to all SGs, then the failure of any feedwater control valve or its respective SOV could prevent feeding all SGs automatically. The inspector brought this contradiction to the licensee's attention on September 11, 198 As a result of several discussions between NRC and licensee personnel, the licensee committed to provide a prompt submittal of information regarding this event, the current AFW system design basis and the licensee's position regarding continued plant operatio This submittal was sent to NRC Licensing on September 20, 198 It will be reviewed in detail by NRC Licensing. This item will remain unresolved pending NRC completion of that evaluation (UNR 50-213/85-20-02).

4.3 AFW Actuation System Wiring Errors

'

Following the 1979 accident at Three Mile Island, the NRC required all licensees to modify their AFW systems to provide automatic 1 actuation of AFW. This modification was to be performed, initially, using control grade components and later upgraded to safety grade specifications. The licensee performed these modifications under project assignment 79-202 and plant design change request (PDCR) No and 401. After the identification on September 10, 1985, of four SOVs cross-wired to the wrong power supply, a review of the implementation and testing of these modifications was conducted to determine the cause of these errors. As stated earlier, the inspector confirmed that the source of power to the SOVs is not critical to the safety function of the AFW actuation circuit. All six SOVs are de-energized upon actuation of either the A or B train AFW trip logic. The significance of the mis wiring is related to the quality of current as-built configuration information. Incorrect system drawings can complicate and delay plant troubleshooting, maintenance, and emergency repair In addition, the quality of future modifications could be compromised by the use of inaccurate design informatio PDCR 384 was implemented in 1980 to provide control grade automatic actuation of AFW. During' this installation, the six S0Vs were installed, piped and wired in the back of the main control boar No definitive documentation of details of the SOV piping and wiring installation or of associated quality control checks were identifie Post installation testing with regard to the SOVs consisted of verification that the system actuated properly on both A and B train

_ _ _ -. _ _ _ _ _ ._- .

,

.

.-

trip Component testing which would identify a power. supply miswiring was not conducted. The plant operated throughout fuel cycle 10 with this control grade system operable..In 1981, the licensee installed PDCR 401 to upgrade the AFW initiation system to safety grade.. Among other modifications, the SOVs were replaced with a qualified design. Again, the record of installation controls does not document details which would substantiate the correct arrangement of power supplies to the SOVs. Quality control records on this project.were also not conclusive, however, many discrepancies were identified by-QC including the absence of labels on the S0Vs and associated wiring. In addition, problems in splicing the S0V electrical leads were numerous and resulted in a decision to rework this wiring to remove unnecessary splices. This rework was postponed to the following refueling outage. TSe post-modification testing for PDCR 401 again was a system functional test with respect to the S0V operations.- No test was performed which would verify component power ,

supplies. The plant operated throughout fuel cycle 11 with the safety grade AFW actuation system operabl During the 1983 refueling outage, the S0V wiring was reworked. Installation procedures and QC records provi.ded no details which would verify the as-built configuration before or after the rework. Testing after -

this rework was done using the routine system surveillance procedure, which could not identify the crossed wiring. No record was iden-tified indicating when the 50V and wire labels were installe The licensee has postulated that the wiring could have been correctly terminated in 1980 and 1981 and that the labels were subsequently mis positioned (reversed for the A and B pump SOVs and the #3 and #4 control valve SOVs) during label installation. The licensee concluded that during the rework of S0V wiring in 1983 the wires were incorrectly terminated in accordance with these mis positioned labels. No one verified the actual' function of the SOVs being connected. This position was supported by licensee recollection that installation controls (rework of only one S0V at a time) were only transmitted orally to workers, and plant operators recall that, in fact, all wires were removed first and then re-installed. Job working and design documents were not sufficient to confirm point-to-point wiring checks to the-SOVs, because the SOV terminations are not detailed on the main control board drawings. The inspector reviewed all of the available design, installation, Quality Control and test documentation associated with these modifications. The following conclusions were reached: The documented installation controls and in process quality control (QC) checks do not support the correct installation of the AFW modification, except in their lack of unsatisfactory QC findings related to wire terminations. The quality of installation controls and in process QC checks were essentially the same throughout this four year perio Post-modification testing emphasized system functional test Component level tests were conducted in actuation logic trains

(

'

-

.

to verify each appropriate actuation trip separately, however, no test which would verify correct alignment of system / component power supplies was identifie Based on the preceding, the inspector could not conclude that other modifications performed using the same program for implementation of modifications could not have similar wiring errors. Such errors should not affect system function under normal operating conditions, but could affect system operation under special conditions requiring the exercise of' certain separate and/or redundant parts of a desig These conditions often present themselves due to system component failures or when systems are intentionally degraded for maintenance or testing. The inspector also noted that, although the licensee's modification control program has recently been upgraded, the requirements and documented guidance as to the appropriate level of specification and documentation of installation controls, in process QC, and a phased approach to post-installation testing has not changed significantly since these AFW modifications. These concerns were discussed with licensee management during the inspection. The licensee has performed detailed investigations regarding the design change control aspects of this event and is in the process of formalizing corrective actions in response to that investigation. The licensee stated that: (1) it is likely that the identified SOV mis-wiring occurred only during the 1983 wiring rework and was primarily a mislabeling problem; (2) a major electrical system as-built verification project has been completed within the past few years; and (3) the efforts of the NRC ordered Plant Design Change Task Group (PDCTG) in reviewing many old PDCRs (including PDCRs 384 and 401) has verified the adequacy of these designs and the adequacy of the post-installation testing from a system functional operability standpoint. The inspector acknowledged the licensee effort 1n this area. The PDCTG reviewed the AFW modifications in detail, but did not perform system wiring walkdowns because their review of the design and post modification tests focused on overall system function. Because the AFW system functioned properly with the mis-wiring, the PDCTG review did not identify the error. The PDCTG has made recommendations regarding the quality of installation and post-modification testing. Since none of the licensee efforts cited above had identified the AFW S0V mis-wiring, NRC could not conclude that no other more significant errors existed in the plant. The licensee indicated that he has confidence that there is a low probability of other such wiring errors in the plan The licensee has not indicated that further review of other plant modifications or circuits will be conducted. The licensee has stated that corrective actions are being evaluated for utility-wide improvements in the following areas:

--

Design Change Installation Controls

--

In process quality control and documentation

--

Phased approach to modification testing

--

Overall control of work onsite

V

-

.

The inspector stated that this issue would remain unresolved pending further review by NRC management (UNR 50-213/85-20-03).

5.0 Unresolved Items Unresolved items are matters about which more information is required in order to determine whether they are acceptable items or violation Unresolved items identified during this inspection are discussed in Paragraph .0 Exit Interview On September 25, 1985, the inspector held a meeting with members of the licensee's management staff concerning the scope and finding of this inspection. No proprietary information was identified related to materials reviewed during this inspectio I I

l . -