IR 05000255/1986017

Augmented Investigation Rept 50-255/86-17 on 860522-25 Re Concerns Over Multiple Equipment Failures.Significant Facts Re 860519 Reactor Trip Identified.Root Cause Determination for Several Equipment Failures Still in Progress

ML18052A592
Person / Time
Site:	Palisades
Issue date:	06/27/1986
From:	Norelius C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML18052A591	List: IR 05000255/1986017 ML18052A590
References
50-255-86-17, NUDOCS 8607100381
Download: ML18052A592 (28)

Text

U.S. NUCLEAR REGULATORY COMMISSION AUGMENTED INVESTIGATION TEAM Report No. 50-255/86017(DRP)

Docket No. 50-255 Licensee:

Consumers Power Company 212 West Michigan Avenue Jackson, MI 49201 Facility Name:

Palisades Investigation At:

Palisades Site, Covert, MI Investigation Conducted:

May 22-25, 1986 Team Members:

Approved By:

C. W. Hehl, Chief Operations Branch, DRS-Rill B. L. Burgess, Chief Section 2A, DRP-RIII T. V. Wambach, Licensing Project Manager, PBD8, NRR e~

/.J. /tc.-----~

C. E. Norelius, Director Division of Reactor Projects Region III License No. DPR-20 V. D. Thomas Engineering & Generic Communications Branch, DEPER, IE N. B. Le Maintenance and Training Branch, DHFT, NRR E. R. Swanson, Senior Resident Inspector Palisades

TABLE OF CONTENTS General Discussions Reactor Trip Review Sequence of Events Plant Parameter Review. Operator Actions....

Equipment Deficiencies Evaluation. Operations Department Interviews. Confirmatory Action Letter Backfit Analysis Persons Contacted. Licensee/NRC Meeting. Conclusions *....

Page

• 3

6

...10

..10

... 23

... 24

• * * 25

..27

... 27

.28

• DETAILS General Discussions Background In the most recent SALP report for Palisades, which covered the period November 1, 1984 through October 21, 1985, the areas of Maintenance, Surveillance, and Quality Programs and Administrative Controls declined from a rating of Category 2 to Category The low ratings in these areas were, in part, due to a lack of aggressive corrective action by the licensee and poor management controls and attitude On October 30, 1985, Region III issued a Confirmatory Action Letter (CAL-RIII-85-15) to the licensee as a result of the licensee's failure to achieve a significant reduction in the maintenance backlog relating to both safety and nonsafety-related equipment; the overall maintenance program and maintenance backlog had been the subject of two prior inspections conducted earlier in 198 Starting in late 1984 and continuing through calendar year 1985, five separate events occurred related to leaking Safety Injection Tank (SIT) check valves despite maintenance on those valves during the Cycle 5 refueling outag During the Cycle 6 refueling and maintenance outage two of eight SIT check valves were rebuil Additionally, the licensee elected not to perform maintenance on Reactor Coolant Pumps (RCP's) with indicated seal pressure oscilla-tions; a condition indicative of incipient seal failur During the week of March 3, 1986, Palisades facility was returning to operation from the Cycle 6 extended refueling and maintenance outage with the first stage seals not staging and pressure oscillations up to 600 psi on the second stage of two of four Primary Coolant Pumps (PCP's).

In addition, valve leakage problems were identified on a primary coolant system loop check valve on the HPSI injection line, two Safety Injection Tank (SIT) pressure control valves and a manual isolation valve, and the three-way divert valve in the chemical and volume control syste A management meeting between Consumers Power Company and the NRC was held on March 6, 1986, to address these operational problems and based on this meeting and a subsequent conference call, the licensee elected to shut down and repair these problems on March 8, 198 After completion of repairs, the licensee returned the unit to power operation on March 25, 198 On March 26, 1986, the reactor tripped in response to a turbine trip resulting from main generator voltage regulator reset switch problem Following troubleshooting, the unit was restarted on March 27, 198 On April 10, 1986, Palisades again shut down after exceeding the Technical Specification limit for unidentified primary coolant system leakag Licensee investigation determined the cause to be a failed relief valve in the letdown subsystem for the chemical

and volume control syste During return to power operation on April 11, 1986, the licensee identified a packing failure on Condensate Pump 11A.

The pump was repacked twice prior to replacement on April 19, 1986, with an onsite spar The plant was maintained at approximately 53 percent power during replacement of the condensate pum Following replacement of the condensate pump the unit was maintained at 53 percent power while the licensee determined the cause of oxygen in-leakage into the secondary syste On April 23 and 29, 1986, existing valve leakage problems in the primary coolant makeup system resulted in primary coolant temperature excursions which, although later determined to be enveloped by the accident analysis values, were initially reported as potentially exceeding primary coolant temperature limitations associated with the main steam line break analysis. Automatic and associated manual valves in the pure water and boric acid subsystems of the makeup water system were leaking by to the extent that, during operator attempts to inject makeup water dilutions or borations, the concentrations of makeup water could not be accurately determined. These valves were repaired a few days prior to the May 19 tri A special task force review conducted by NRC Region III and NRC contractors during the period January 10 through March 7, 1986 identified general weaknesses with respect to recurrent equipment problems requiring significant operator attention, procedural adherence, quality of procedures and timeliness of procedure changes and thoroughness of reviews and scope of corrective action. These weaknesses were transmitted to the licensee in a report dated May 16, 198 On May 19, 1986 the plant experienced a reactor trip from 97 percent power in response to a high pressurizer pressure conditio The high pressurizer pressure condition was a direct result of a loss of control power to the turbine electrohydraulic control system causing the turbine governor valves to clos Following the reactor trip, during plant stabilization numerous additional equipment failures occurre As a result of these events, on May 21, 1986, the NRC reiterated concerns to licensee management over the potential for serious challenges to safety systems that these events pose, and the burden failures of this type place on the Operating Staf Licensee commitments made in response to these concerns were confirmed by a Confirmation of Action Letter (detailed in Paragraph 5 of this report) sent to the licensee on May 22, 198 Additionally, on May 22, 1986 the Augmented Investigation Team was dispatched to the Palisades site to conduct an independent investigative review of the equipment failures that occurred during the May 19 trip and to determine the impact of these failures on the plant operating staff.

• May 19, 1986 Reactor Trip On May 19, 1986 the Palisades Plant was operating at approximately 97 percent of full powe The unit had been at power for over a month following restart from a forced outage on April 10, 1986, to repair a leaking relief valve in the letdown syste Shortly before 2:16 p.m. (EDT) on May 19, 1986, both primary and secondary Electro Hydraulic Control (EHC) power supplies tripped, causing a loss of EHC control power and allowing the main turbine governor to valves to drift closed. This loss of turbine load initially resulted in a Reactor Average Temperature (Tave) increas (It is noted that, at the time of this event, the plant was operating at an elevated average temperature to maximize electrical outpu This elevated Tave was necessitated by the number of plugged tubes in the Steam Generators.)

The increase in Tave was promptly identified by the primary plant control room operator (C02) who immediately initiated manual rod insertion. Simultaneously, the control room turbine operator (COl) observed loss of governor valve indication and based on his evaluation of plant conditions reached to trip the turbine. Prior to operator initiation of turbine trip, the reactor tripped on high pressurizer pressur The reactor trip occurred at shortly after 2:16 p.m. (EDT).

Following the reactor trip, the operators initiated post trip response in accordance with Emergency Operating Procedure EOP-Although not required by the emergency procedure, the operator followed the automatic trip with a manual reactor tri During a scan of the control boards, the operator noted the Turbine Bypass Valve (TBV) and one of four secondary system Atmospheric Dump Valves (ADVs) failed to ope These failures caused the lifting of banks of Main Steam Safety Valves resulting in a rapid decrease of Steam Generator level and a corresponding cooldown of the primary syste To limit primary system cooldown, the operator tripped both Main Feed Pump The Motor Driven Auxiliary Feedwater Pump initiated automatically to maintain Steam Generator Level In an attempt to more rapidly restore Steam Generator Levels to normal, the Turbine Driven Auxiliary Feedwater Pump was manually started, causing a further cooldown of the PC During observation of PCS parameters, the operator observed that a pressurizer spray valve indicated ope Noting that pressurizer pressure had decreased to a value at which the valve should have been shut, and the continued decrease in pressurizer pressure towards the Safety Injection Setpoint, the operator attempted to manually start the third charging pum Despite repeated unsuccessful attempts to start the third charging pump, pressurizer pressure bottomed out at 1689 psia, which is above the SI injection setpoint of 1593 psia, and immediately started to recove *

During subsequent plant stabilization and recovery, the operators noticed that control rod 34 rod bottom light was not illuminate Comparison of the primary rod position indication to the secondary indication system determined that all rods had inserte Following return of pressurizer level to above the letdown isolation setpoint, the operators attempted to restore letdown flo During restoration the letdown orifice back pressure regulator (CV2012)

failed closed, requiring the operators to shift to the redundant regulato The redundant regulator functioned properly, although, it had previously been identified as an item tagged for maintenanc The plant was stabilized in Hot Shutdown pending post trip review and evaluation of equipment failure The licensee notified the Nuclear Regulatory Commission, pursuant to 10 CFR 50.72, of the plant trip. During the initial notification at 2:54 p.m. EDT May 19, 1986 the licensee identified the trip initiat-ing signal as loss of turbine load turbine trip. Further evaluation by the licensee determined the initiating Reactor trip signal to be pressurizer high pressure. This clarification was subsequently made to the NRC via the EN Subsequent review by the licensee of their emergency plan resulted in a declaration of an Unusual Event (UE).

The UE was declared and terminated at 5:45 p.m. ED The NRC HQ Operations Center was notified of the UE at 5:52 EDT on May 19, 198 (It is noted that during each ENS telecon between the plant and the NRC operations center, the NRC duty officer asked the plant if it was a normal trip and if everything functioned properly; each time the plant responded that the trip was normal and everything functioned properly). Reactor Trip Review Sequence of Events The following edited sequence of events was derived by review of the plant TENNECOMP DATA LOGGER PRINTOU The editing consists of not reporting here the many other data points recorded that were not considered significant in the abnormal responses of some of the equipment in this event and rounding the times recorded from milli-seconds to the appropriate values of significanc NOTE:

Information enclosed in parentheses is not from the data logger but is added for informatio /19/86 14:15:14 Main Turbine speed Low (Turbine Governor Valves closing).

14:15:51

15:54

16:01

16:09

16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:1 :16:2 :16:25

16:27

16:2 :16:33 Steam Generator A and B pressure hig Pressurizer Spray Valve CV-1057 ope Pressurizer Level Hig Letdown orifice Stop Valves CV-2004 and 2005 opened (plant was operating with the third stop valve CV-2003 open).

Pressurizer pressure Channels B and D pressure hig Reactor Control Rod Drive clutch A, B, and D Power Relays K-1, K-4, and K-3 deenergized; Pressurizer Pressure Channel A pressure high; Reactor trippe Turbine Trip Relays K-305R and K-305L tripped (Turbine Trip) Pressurizer Pressure Channel C pressure High (Last of four channels).

Safety Channel D-Upper neutron flux lo Safety Channel 0-Lower neutron flux lo Turbine Stop Valves CV-0569 and 0573 close Pressurizer Pressure Channel C pressure norma Pressurizer Pressure Channels A, B, and D pressure norma Turbine Stop Valve CV-0571 closed (third stop valve).

Safety Channels A, B, C-Upper neutron flux lo Safety Channels A, B, C-Lower neutron flux lo Atmospheric Dump Valve CV-0782 ope Auxiliary Feedwater Pump (AFW) P8A Started (Manual)

Feedwater Pump P-lA Drive Turbine trippe Pressurizer Level norma Letdown orifice Stop Valves CV-2004 and 2005 closed (CV-2003 still open).

Charging Pump P-55B started (P-55C already running).

Atmospheric Dump Valve CV-0781 open (second ADV).

• 14:16:36

16:3 :16:49

16:5 :16:5 :16:5 :17:01

17:0 : 17: 10

17:2 :17:57

18:29

18:32

19:03

19:07

19:38

19:46

20:17

22:15

22:17

22:52

22:5 :23:23 Turbine Stop Valve CV-0575 closed (fourth and final stop valve).

Pressurizer Spray Valve CV-1057 close Pressurizer Backup Heaters OFF (Pressurizer level 34%)

Charging Pump P-55A started (Manual start by RO, P-55 Band C already running).

Pressurizer Level Lo Letdown orifice Stop Valve CV-2003 close (CV-2004 and 2005 already closed at 14:16:27).

Feedwater Pump P-lB Drive Turbine trippe Pressurizer Pressure Norma Atmospheric Dump Valves 0781 and 0782 close Charging Pump P-55A stopped (tripped not manual stop)

Charging Pump P-55A started {manual start by RO).

Charging Pump P-55A stopped (tripped not manual stop).

Charging Pump P-55A started (manual start by RO).

Charging Pump P-55A stopped (tripped not manual stop).

Charging Pump P-55A started (manual start by RO).

Charging Pump P-55A stopped (tripped not manual stop).

Charging Pump P-55A started (manual start by RO).

Charging Pump P-55A stopped (tripped not manual stop).

Pressurizer Level Norma Atmospheric Dump Valve CV-0781 ope Pressurizer Back-up Heater o Letdown orifice Stop Valve CV-2003 opene Charging Pump P-55A started (manual start by RO).

14:23:28 Charging Pump P-55A stoppe (tripped not manual stop)

35:21 Charging Pump P-558 manually stoppe :38:37 Turbine Driven AFW Pump P-8B o Discrepancies or peculiarities noted in review of the data logger printout included the following:

(1)

Reactor Control-Rod Drive Clutch C Power Relay K-2 did not record as deenergized for reactor trip. All rods were verified to insert by the operators following the tri (2)

Reactor Load/Turbine Trip Channel B did not record as did Channels A, C, D at 14:16:11.11 and 14:15:1 (3)

Turbine Stop Valve CV-0571 indicated closed three seconds after Stop Valves 0569 and 0573 which closed.3 seconds after indication of turbine trip and Stop Valve CV-0575 indicated closed 25 seconds after indication of turbine tri (4)

Neutron Flux Channel D indicated response to reactor trip three seconds faster than Channels A, B, and (5)

Atmospheric Dump Valve 0780 did not indicate opening on Data-logger; however, was indicated opening and closing on Critical Functions Monitor and by indications in the control roo Plant Parameter Review The following information was derived by review of selected strip charts and the printout of the data-logge Reactor coolant temperatures, pressure and pressurizer level increased during the load rejection caused by the turbine throttle valves closing. Pressurizer pressure increased from 2020 psi to 2245 psi The hot leg temperature increased from 588°F to 594° The cold leg temperatures increased from 539°F to 557° Pressurizer level increased from 56 percent to 69 percen The range of the pressurizer water level channel is 100.7 inches to 236 inches from the bottom of the pressurizer which is 336 inches in heigh On the secondary side, steam generator pressure increased from approximately 730 psia to 1025 psi The steam generator water levels were stable at 70 percent until steam generator pressures started rising and the level started decreasin Following the reactor trip, the void collapse dropped the steam generator levels to a minimum of about 12 percen The range of the S/G water level channel is from approximately 395 inches to 575 inches from the bottom of the steam generator which is 709 inches in heigh The

• steam pressure dropped to approximately 860 psia at which point the atmospheric dump valves close Then pressure slowly rose to approximately 960 psia where it was controlled by an atmospheric dump valv The primary system response after the trip was normal except for the back-pressure control valve in the letdown line that failed closed and the pressurizer spray valve that didn't close completel However, those two deficiencies did not significantly affect the primary system parameter The pressurizer pressure dropped to a minimum of 1689 psia (SI low pressure setpoint 1593 psia) at minutes after the reactor trip. Then it was on a controlled slow increase reaching 2000 psia, one hour later. Pressurizer level dropped to 26 percent within the first 1.5 minutes and was then returned to the programmed level setpoint of 42 percent about 15 minutes later. Primary coolant temperatures dropped to 535°F in about 3 minutes and then slowly rose to a stable 540° Operator Actions At the time of the reactor trip, the control room was staffed by a Shift Supervisor (SRO), a Shift Engineer (SRO), and two Control Operators (RO).

This staffing met facility license requirement The operators apparently had no prior indication of a proble Based on reviews of data from the trip and interviews with the on-shift personnel, it is the opinion of the AIT that operator actions during and following the May 19, 1986 trip event were prudent and correct. However, it is also apparent that equipment problems that were either present before or occurred during the event added significantly to the burden placed on the operator.

May 19th Eg~ment Prob 1 ems The team conducted an in-depth evaluation of the Equipment problems identified during the May 19, 1986 trip. These evaluations included a review of the licensee root cause diagnostic activities, corrective action and post-maintenance testing with regard to the current problem The team also reviewed the maintenance histories of these component Pressurizer Spray Valve During the May 19 reactor trip event, Pressurizer Spray Valve CV-1059 indication lights were observed by operations personnel to have an "open indication" on the control panel, but, at the same time, the pressurizer pressure was observed to recover normally during the even Palisades plant maintenance staff performed an evaluation to determine the actual valve action during the pressurizer pressure recovery perio From interviews with the licensee maintenance staff, the AIT has learned that the licensee has initiated a work request to allow troubleshooting of the Pressurizer Spray Valve,

CV-105 The licensee system engineer stated that prior to trouble shooting, the valve was observed not seated in the fully-closed position. During troubleshooting, the air supply line was disconnected from the valve positioner, and the valve was observed to move to its "fully-closed" positio When the air line was reconnected, the valve was observed to have moved slightly from its

"fully-closed" positio Further troubleshooting, indicated that a back pressure of 5 psig existed in the open nozzle of the valve positione In addition, the AIT also learned that the support bracket and its associated housing for the "valve closed indication~ limit switches were bent in the downward position and may not have allowed the lever arm to be switched when the valve closed initiall A review of maintenance data for this valve indicated that the valve position indicator was adjusted, the valve actuator was disassembled, and the valve was repacked in February 198 Root Cause:

Maintenance staff involved with the valve performance assessment indicated that the bracket supporting the lower limit (position)

switch box was found to have corroded and bent downwar This bracket was mounted relatively near the packing gland where boric acid was reported to have leaked and deposited around the packing and surrounding are The upper bracket supporting the upper limit (position) switch box was found to have no corrosion and was observed to be structurally stable. The root cause of the switch failure was the repeated packing leaks that contaminated and corroded the bracket and the poor practice of not cleaning up leakage residu On May 25, 1986, a licensee maintenance staff member informed the AIT that upon examination of the pressurizer spray valve, the input air control pressure to the valve positioner was reading 5 psi and not 3 psi as it should have for a "Fully-Closed" valv The valve was calibrated to operate between the fully-closed and fully open positions over a 3-15 psi control signal rang To support the conclusion that the valve may have remained partially (unseated) open, the valve was observed to reach a fully-closed position when the air input line was disconnected, depressurizing the valve actuato The root cause of this condition was under investigation by the licensee, but may be related to a previous contamination of the air system by desiccan Maintenance History The maintenance record on CV-1059 showed that a total of seven Work Requests (WR) were written during period period from October 27, 1983 thru February 27, 1986 with time intervals varied between each as follows:

October 27, 1983 July 19, 1984

August 5, 1984 October 17, 1985 October 18, 1985 February 25, 1986 February 27, 1986 The predominant problems with the valve, as indicated from these WRs, were packing leakage and, at one time (October 17, 1983) desiccant was found in the equipment air lin The review of the valve maintenance records (i.e., Work Order packages) resulted in the following observations:

No diagnostic activities were evident in the record Corrective actions on the packing leak were either adjusting the packing, or replacing the packin No evidence was found to indicate any evaluation of the effect of the leaked boric acid may have on material and related valve mechanisms surrounding the valve packin Performed work did not fully address the problems described in the Work Request It was not evident that either the initiators of the Work Requests or the System engineer were involved in the review of completed wor Post maintenance testing and operability testing requirements were not always detailed enough to demonstrate that the valve would perform to meet its design requirement Pr~B~~~d Corrective Action:

Work Requests have been written to allow equipment assessment and repai Interviews with the licensee maintenance staff indicated that the lower bracket supporting the switch box will be replace Limited corrective work on the stem position arm and repositioning of the lower switch box were completed to permit correct valve position in-dication in the control roo The pneumatic control will undergo required maintenance to obtain the proper span of 3-15 psi Chemical and Volume Control System The chemical volume control system (CVCS) primary intermediate back pressure regulator control valve (CV-2012) failed closed. This condition resulted in a rapid pressure rise in the letdown system to where the relief valve was activated.

As a result of the above malfunction the licensee issued Work Request No. 086925 on May 20, 1986 to plant maintenance personnel for resolution of the proble *

A study of the CV-2012 malfunction by the AIT included a review of the Work Order (WO) history for CV-2012 and associated components which comprise the eves intermediate back pressure regulating syste These components include eV-2122, the standby pressure regulator valve, and PIC-0202, the automatic indicating controller for valve positioning of CV-2012 or CV-2122 when either valve is selected for automatic operatio The AIT study of the CV-2012 valve malfunction was expanded to include discussions with knowledgeable plant personnel who have direct responsibility to review and correct malfunctions of components within the eves as they aris Based on the above review and discussions, the following findings were mad On April 16, 1986 WR 082586 was written to address a problem with Pie-0202 (would not control in Automatic).

WO No. 24605245, initiated on May 2, 1986 to resolve this problem with Pie-0202, was implemented during the week of May 12, 198 A static system test of the afore-mentioned problematic components in the eves was conducted in an attempt to return them to operatio On May 12, 1986, Operations personnel encountered a problem with back pressure regulator valve eV-2122 and issued Work Request 080336 to identify the cause for the failur The ongoing static system test continued in an attempt to resolve earlier problems encountered with PIC-020 The static testing was completed on May 15, 1986 by WO 2460524 However, dynamic testing/turning of these components needed to be conducted prior to closing out WO 2460524 At this time plant conditions were such that the dynamic testing planned for the eves was delaye Therefore; at the time of the May 19, 1986 event, back pressure regulator valve CV-2012 was in operation and CV-2122 and Pie-0202 were tagged for problem identificatio A review of the work order history and the work requests issued on CV-2012, PIC-0202, and eV-2122 show the following:

Component Work Reauest CV-2012 WR 086925 (May 20, 1986)

CV-2122 WR 080336 (May 13, 1986)

PIC-0202 WR 082586 (April 16, 1986)

Problem Failed Closed (During Event)

Defective Valve - Positioner (Tagged out for problem identificatio Would not control in Automatic In addition to the above status of essential components of the eves, several other WOs and work requests had been issued on these same components since February of this yea *

For example, WO 24503030 was issued on February 20, 1986 because of excessive full open, full close cycling problems experienced with CV-2012 when controlled manually from the control roo The cause for the excessive oscillations was attributed to a weak spring in the valve positioner and an accumulation of dirt in the valve positioner control air port Following a spring change out and cleaning of the valve positioner air ports on February 26, 1986 the valve was returned to servic Less than one month later on March 21, 1986, another WO, No. 24604585 was issued to resolve another problem with the same valv The electric to pneumatic converter (E/P) which is located adjacent to the valve positioner needed repair because CV-2012 would not open or close when the valve control station was put in manual or automatic operatio Following the repair to the E/P device CV-2012 was recalibrated on April 4, 1986 and returned to service as stated abov eV-2012 failed closed during the May 19, 1986 plant tri In discussing the chronic problems that this licensee has experienced with the intermediate back pressure regulating system, plant personnel stated that an extensive study is underway to determine if a redesign of certain sections of the eves, replacement of the old subcomponents (e.g., valve positioner, trim valves, letdown orifices, etc.) and possible relocation of the pressure sensor would result in alleviating the system instability problem AIT review of the plant post trip data report indicated that the letdown orifice stop valves eV-2003, eV-2004, and eV-2005 closed when pressurizer level decreased below norma At this time, it appears that back pressure regulator valves ev-2012*should have been close When eV-2003 reopened after pressurizer water level returned to normal, back pressure regulator valve eV-2012 failed closed and the eves system relief valve lifted on excess pressur Pressure control was then shifted to eV-2122 maintained normal intermediate pressur Information on the open/close positions of eV-2012, eV-2122, and the eves letdown stop valves are noted in the 11 Post-Trip Data Report.

11 Atmospheric Steam Dump Valve During the May 19 event, Atmospheric Steam Dump Valve eV-0779 did not open in response to steam line high pressur The remaining three steam dump valves eV-0781, eV-0780 and eV-0782 functioned properl Palisades plant maintenance staff has performed a preliminary investigation as to why eV-0799 di~ not function during the reactor trip even The maintenance staf~ indicated that there may be 11miswiring 11 between the remote shutdown panel e33 and the individual control mechanism of valves eV-0779/0781.

It was also found that glycol had leaked out of the valve operator; however, the plant maintenance staff has determined that this leakage alone would not have prevented valve operatio The review of maintenance data indicated that valve CV-0779 was last worked on by the maintenance staff on March 11, 1986 to correct valve packing leakag The valve was repacked with high temperature/

~igh pressure Chesterton packing and new studs were also installed in the bonnet An operability test was performed and accepted by Operations on March 22, 198 On February 20, 1986, work was also performed under WO No. 24603223 on CV-0779 following an event where the valve failed to stroke during performance testing. Maintenance data from this work revealed 11wires going to E/P 0779 and E/P 0781 on wrong terminals 11 and maintenance personnel 11 reversed the wires to the E/P 1s.

The work order package for this activity contained no references to a wiring/connection diagram and no detailed documentation on how the wires were reversed or how the rewiring compared to the original design drawing Post maintenance testing was indicated as performed; however, no details were give Based on this testing the valve was declared operable on February 26, 198 During troubleshooting while the AIT was onsite, CV-0779 would not stroke from the remote panel until wires were swapped at the controlle Root Cause:

Based on interviews of the involved plant maintenance staff and previous maintenance activities the AIT members have found that the maintenance activities during the month of February 1986 and other ongoing activities may have contributed to the CV-0779 failure to open on high steam pressur The involved plant staff indicated that the cause of the failure is under further evaluatio Maintenance Histo.!'.'_Y:

Data packages for CV-0779 were reviewed for all maintenance activities that occurred from September 15, 1983 through March 10, 1985 and consist of the following activities:

September/1983 - Valve was replace March/1984 April/1984 July/1984

- Solder joint in air line was broken at base of the accumulato Two valve positioner gauges were replace Steam dump system controls.

• October/1984

- Replaced packin February/1985 - Adjusted packin March/1985

- Replaced two damaged vent line The review highlights the following concerns:

The lack of the licensee 1s preplanning, control of diagnostic activities and providing technical guidance during preplanning and performance of diagnostic activities. This exemplified during the troubleshooting to determine the reason why 11CV-0779 did not stroke 11 during the post maintenance testing activities as documented in WO No. 2460322 The lack of adequate post maintenance testing requirement This is exemplified in the testing of CV-0779 from the HIC-0781B at the C-33 cabinet in that the testing activities did not indicate whether it was required that the test of CV-0779 be carried out from the main control room (WO No. 24603223).

The lack of technical review of the completed maintenance work package to determine the adequacy of work performed and to ensure that post maintenance testings have been carried out to fully demonstrate that, once tested, the equipment reliability and operational readiness are assured. This was exemplified in WO No. 2460322 Proposed Corrective Action:

The licensee has not yet proposed any corrective action to resolve the valve failure; however, a work order has been initiated to troubleshoot and repair of the steam dump control syste A complete I&C check of the valve 1s control system is planne Turbine Bypass Valve The turbine bypass valve, CV-0511, did not open as designed during the May 19, 1986 reactor trip even CV-0511 was operating on automatic control at the time of the even Following the reactor trip, the control room operator discovered that CV-0511 had not opene The operator tried to open CV-0511 by manual control from the control room, but this attempt faile The control room operator subsequently controlled secondary system steam pressure by using automatic and manual control of the atmospheric dump valve As a result of the above valve malfunction, the licensee issued a work request on May 20, 1986 to plant maintenance personnel to study the CV-0511 malfunction.

According to the licensee, plant maintenance personnel are currently conducting studies involving measurements, tests, and observations to isolate the root cause for failur Root Cause In discussing the valve malfunction with the maintenance staff member involved with the current valve performance assessment, the AIT found that some significant changes were made to CV-051 The valve had not consistently performed its intended function of automatically or manually controlling significant changes in secondary steam pressure following a plant trip or load reduction for an extended period of tim As recently as February 27, 1986, WO No. 24604018 called for CV-0511 to be repositioned from a 45° angle mount to a vertical mount position. This WO shows that the 400-lb spring in the valve operator mechanism was replaced with a 1700-lb valve position spring. According to the maintenance staff, this replace-ment was based on the manufacturer's recommendatio WO No. 24604018 was completed March 20, 198 In addition, the AIT was told that the packing bushing was badly galled and required machining to allow it to fit. Subsequent testing of CV-0511 showed difficulty in automatically controlling the valve to various intermediate positions between full close to full ope The valve was repacked to stop steam leaks by WO No. 2460469 This was completed on April 12, 1986. Three days later WO No. 604750 showed that the licensee recalibrated CV-0511 because it did not control steam pressure following a reactor trip. Subsequently, CV-0511 failed to open automatically or manually during the May 19, 1986 reactor trip even The root cause for the recent May 19 failure of CV-0511 has not been identified. The licensee is currently conducting an extensive review to isolate the cause for these chronic valve failure Maintenance History The following work orders issued since October 1985 indicate areas where maintenance was performed to correct the valve malfunction WO Date of Completion 24501828 10/18/85 24505607 02/04/86

Action Installed filter in air supply to CV-0511 (erratic control).

Dial indication test for possible bent valve stem (erratic control).

j *

24603105 03/15/86 24604018 03/20/86 24604694 04/12/86 24604750 04/15/86 Valve would not open/close during test (RR-1-) (cause not identified).

Repositioned valve mounting from 45° to vertical, plus rebuilt the operato (Note:

CV-0511 failed to control during reactor trip event on 3/26/86).

Repacked CV-0511 to stop steam leaks (no mention of recalibration).

CV-0511 failed to control pressure follow-ing reactor trip-reca l ibrate Although there were work orders and work requests issued in the past regarding problems with CV-0511, our review focused on the six WOs discussed abov The review indicates that dirty air in the air system, replacement of the spring in the valve operator, and the method of repacking the valve all could be contributing causes for valve malfunctions. Concerning the replacement of the valve spring, the four atmospheric dump valves had the same design characteristics and similar application as CV-0511 and are presently functioning with the 400-lb spring similar to that which was removed from CV-051 The AIT concluded that (1) the complete air system at the Palisades Plant should be examined to identify the cause of poor air quality, (2) the 1700-lb spring replacement should be studied to determine its effects on the characteristics of the valve operator/control and (3) the method of performing maintenance on valve packing should be reviewe Proposed Corrective Action The licensee has issued a work request to assess valve performance under static test condition If these tests are successful, CV-0511 will then be tested under operating condition EHC Power Supplies At the time the AIT team arrived at the site, the cause of the loss of the EHC power supply was not determined with any degree of certainty. The AIT was informed that, at the time of trip, plant I&C was performing a preventive maintenance (PM) on the cooling fans/

filters in the EHC cabinet located in the turbine buildin The PM activities consisted of (1) cleaning the fans/filters, and (2)

verifying operation of the fans. There are two fans installed on the cabinet doors to provide cooling to the EHC cabinets (3 cabinets).

Air is drawn from outside the turbine building through the cabinets and out through these fan Troubleshooting by licensee staff on May 19, after the reactor trip, indicated that both of the +15 Volt DC power supplies (P/S) had tripped due to the overvoltage protection unit Power input to the primary +15 Volt DC P/S is 120 Volt, AC, 60 Hz from a nearby cabinet (C-21), and power input to the secondary +15 Volt DC P/S is 200 Volt, AC, 420 Hz from the standby permanent magnet operator (PMG) sourc The licensee's troubleshooting found that the unplugging of the 11front door 11 fan would cause the primary P/S to trip on overvoltage protection; however, the secondary P/S would not trip by the same unplugging action nor by using a radio frequency source. A subsequent bench test on both P/S 1s found nothing damaged or malfunctioning and they were returned to the cabine Other checks performed by the licensee during troubleshooting of the P/S 1s were:

All diodes were checked and were found to have functioned normall A loose ground connection was tightened and was found to have no effect on the tripping of the P/S 1 A blown fuse was found and was replaced (200V, 420Hz power type).

The reason for the blown fuse and the time when the fuse did blow were not know From a discussion with the licensee's maintenance engineer, the AIT members learned that these two fans may have been added to the EHC cabinets some time in 1981 when the original power supplies were replaced with the existing Lambda P/S units. It appears that the electrical load from these two fans and the input power to the primary P/S were wired to the same 120 Volt, 60 Hz breaker without any analysis as to what effect these fans may have on the performance of the Lambda P/S 1 Following the May 19 trip the licensee disconnected the wiring feeding power to the fans from the breaker (BKR No. 2) that feeds power to the primary P/ The fans* wiring was reconnected to another breaker (BKR No. 3) feeding the existing lights in the EHC cabinets. This was done to determine if a spike from another AC source would affect the tripping of both P/S's. Again, the plug for the front door fan

was unplugged, and this time, neither P/S 1s trippe *

On May 20, the licensee decided that the turbine speed should be brought to 1200 rpm to check out what effect the AC spikes would have on the 420 Hz power supply. This turbine speed was achieved to generate the PMG 200 Volt, 420 Hz source input to the P/ Again, the licensee was able to trip the primary P/S, and not the secondary P/S by pulling the plug on the front fa The system was left running overnight, powered from the secondary P/ On May 21, the licensee once again tried to trip both primary and secondary P/S's, using the external AC transient but could only trip the primary P/ On May 22, the licensee removed both P/S's to the lab for further testin The results of this testing were as follows:

The licensee was able to trip the secondary P/S with external noise spikes:

It was found that the external spikes have actually caused the voltage to spike, thus tripping the Over Voltage Protection (OVP) uni The OVP unit on the primary P/S was replaced, and it was found that the P/S still tripped on external voltage spike On May 23, the licensee contacted an engineer at Lambda, the vendor for the P/S's, and obtained the following information:

The Lambda engineer verified that the transient (voltage spike)

caused by unplugging a fan on the same circuit with the input power to the P/S could cause a spike in the output of the P/S which would in turn trip the OVP uni The Lambda engineer was not aware if any of the Lambda P/S units would have protection (filtering) against this type of spik The Lambda engineer recommended that the fans be taken off the circuit feeding the P/S' With this information, the licensee decided to recreate the same conditions in the EHC cabinet as existed at the time of the May 19 reactor trip. It was determined that, even though the secondary P/S is not on the same circuit as the fans, the P/S wiring was routed in such a way that it would be affected by any voltage induced from the rear door fan circui The licensee was able to trip both primary and secondary P/S's when unplugging the fan on the rear door of the cabinet.

On May 24, the AIT team was informed by the licensee that the test performed on May 23 on the unplugging of the fan on the rear door was confirmed by another test performed by a team of licensee laboratory test personnel from Jackso Maintenance History Discussions with the licensee maintenance staff indicated that there were no previous failures of the Lamda Power Supplies including from noise induced by electrical disturbances or lightin Root cause The licensee has concluded that the root causes were the common power supply for the EHC cabinet fans and the primary EHC P/S and the close proximity (same cable bundle) of the rear fan power cables to the secondary EHC P/S wiring. This circuit interaction was demonstrated by:

(1) unplugging the fan on the 11front door 11 of the EHC cabinet will only trip the primary P/S; and (2) unplugging of the fan on the 11 rear door 11 of the cabinet will trip both the primary and secondary power supplie Proposed corrective action:

The licensee has proposed to reconnect the two fans to another circuit breaker (BKR No. 3). This breaker is currently used to supply power to the cabinet light In addition, the power wiring for the fans will be rerouted and separated from the P/S wirin Conclusion:

In reviewing the sequence of testing and troubleshooting to determine the root cause of the P/S 1s tripping, the AIT members observed that:

The licensee had initially confined the troubleshooting to the assumption that only one fan was the cause when in fact both fans were unplugged during the performance of the PM on the EHC fans and filter There is no indicating light to indicate which of the P/S 1s is in operation at the EHC pane The AIT team members noted that one indicating light is provided on the turbine control panel in the control room; this light will be on if the primary P/S in the EHC cabinet is lost and the secondary P/S is in service or if the secondary P/S is lost while the primary is in servic However, if the indicating light fails, there would be no indication of subsequent power supply failur The AIT considers this to be a design weakness that should be reviewed by the licensee. It is noted that if both power supplies fail,

j

• the entire turbine control panel goes dark as was immediately detected by the operators. However, insufficient time was available during the May 19 event for the operators to take corrective measure Charging Pump P-55A During the post-trip events of May 19, 1986, charging pump P-55 A, the variable speed pump, was started six times to assist in maintaining pressurizer water level. All six times it tripped after running 30 second The pump was not in service at the time of the event and was considered inoperable because of a cracked block, but had been considered by the Operating Shifts as available for emergency us Root Cause The pump has two 30 second trips, one from low pump suction pressure and one from low lube oil pressur The oil pressure was suspect because of low lube oil pressure reported in April 198 At that time, the corrective measure was to close the recirc valve of the lube oil pump (a gear pump driven from the pump shaft). This recir valve sets the pressure that would be established by the lube oil pum The pump was tested on May 2 and the oil pressure was 25 psig at 617 rp The AIT was told that this was high enough to clear the trip (9 psig) but was not as high as normal (35-40 psig).

The root cause of the loss of oil pressure was not determine The licensee stated that a representative of the pump vendor is coming to the site to assist in troubleshooting the proble Maintenance History Work order listings since 1983 were surveyed and those that involved work that was done on the oil system were reviewed to determine whether this was a developing problem or whether it first occurred in Apri WO 24602108 24502941 24504419 24504863 Date of Completion 10/4/84 9/27/85 9/27/85 10/12/85

Action Tighten plug/stop oil leaks Stop oil leaking from crankcase & possibly shaft seal Fix leaking oil line to oil filter canister Stop packing leaks and oil leaks

• 24601000 24604788 3/16/86 4/29/86 Change oil and oil filter Low lube oil pressure 18-20 psig at 600 rpm, normal 35 to 40 psi Adjusted pressure relief valve to highest pressure possible. Test at 617 rpm produced 25 psi In addition, preventative maintenance procedures were run periodicall However, the AIT could find no indication of any previous loss of oil pressure for this pum Proposed Corrective Action Undetermined at this time. A pump vendor representative is to visit the site to assist in troubleshootin Other Failed and Out of Service Equi£!!1~~~

The AIT reviewed the maintenance history packages for the Rod 34 rod bottom light and the Condensate Pump recirc valve (CV-0730) and determined that the rod bottom light did not pose a significant maintenance concern or hinder plant response to the tri The Condensate Pump recirc valve was required to be repositioned during the event, however, the valve did not significantly alter plant response nor did it require undue operator attention during the tri No maintenance concerns relative to this valve were identifie.

Operations Department Interviews As delineated in Paragraph 1 (above), the recent history of equipment problems coupled with those equipment failures identified following the May 19 reactor trip raised significant concerns regarding the potential burden failures of this type place on the operating staf To gain perspective regarding this concern, the AIT interviewed two Shift Supervisors (SRO), two Shift Engineers (SRO), five control operators (RO), one licensed auxiliary operator (RO) and four members of operations managemen The results of these interviews are as follows:

With regard to the May 19, 1986 reactor trip, interviews with the operators and shift supervisor on duty at the time of the event, verified that the equipment failures identified during the licensee's post trip review were accurate and plant response was expected considering these failures. These interviews additionally determined that the equipment failures and existing out of service/

degraded equipment did divert operator attention during plant stabilization and recovery efforts following the tri.

All operators and shift supervisory personnel except one expressed concern over plant managements decision to start the plant up from the recent refueling outage with the equipment deficiencies existing at that point in tim The operators perceived undue emphasis on terminating the outage and getting the plant back on lin In general, operation personnel felt that there had been significant improvement regarding equipment deficiencies following the March, 1986, mini outage, but there was concern regarding the number of recent equipment failure The operating shift and operations management expressed concern regarding the adequacy of maintenance activities. The operators recognized the need for improving their problem descriptions on maintenance work orders (to aid in problem diagnostics), but noted that the number of reworks required to fix a problem reflected the quality of the maintenance work performe All operators expressed a need to drastically improve communications between operations and maintenance and within the operations organiza-tion (mainly feedback from management).

The operators expressed a perception that 11if it (an equipment failure or degradation) did not jeopardize megawatt production, it didn't get fixed.

The operators expressed the view that the existing physical condition of the plant resulted from implementation of a 11least cost 11 corrective maintenance approach (patch rather than replace)

and that certain pieces of plant equipment (balance of plant) are just worn ou The operators expressed a lack of confidence in the existing plant managemen Operations management interviewed was aware of this lack of confidenc Confirmator~ Action Letter A Confirmatory Action Letter (CAL) from James G. Keppler (NRC-Region III)

to Dr. F. W. Buckman (CPCO) was issued on May 21, 198 The CAL expressed NRC concerns regarding multiple equipment failures that occurred during the May 19, 1986 trip. The CAL required that the unit would be placed immediately in cold shutdown and maintained at or below hot standby conditions (reactor not in startup or critical) until a thorough investiga-tion into the causes and implications of the May 19, 1986 reactor trip and a thorough investigation of plant safety systems and systems important to safety is complete Following these investigations, the CAL required the utility to brief the Regional Administrator, or his designee, on the results of the investigation and the corrective actions taken or planned and obtain the approval of the Regional Administrator prior to restar On May 22, 1986 the unit was placed in cold shutdown and a licensee independent investigation team was initiated to review the May 19 event and to investigate the equipment failure.

Backfit Anal~sis On May 21, 1986, Region III directed the Palisades facility to shutdown pending completion of an investigation into the cause of the May 19 reactor trip and subsequent equipment failures. Subsequently, Region III issued the confirmatory Action Letter described abov Pursuant to NRC Manual Chapter 0514 11 NRC Program for Management of Plant Specific Backfitting of Nuclear Power Plants, 11 Region III prepared an evaluation setting forth the justification for this action. This evaluation which was forwarded to the licensee on June 19, 1986 is presented belo Background EVALUATION OF NRC STAFF IMPOSED BACKFIT NECESSARY TO ENSURE THAT THE PALISADES NUCLEAR POWER FACILITY POSES NO UNDUE RISK TO PUBLIC HEALTH AND SAFETY On May 19, 1986, the Palisades nuclear reactor tripped from high pressure after a loss of turbine control power resulted in closure of the turbine governor valves. Although the plant responded normally, several components did not operate as expecte The turbine bypass valve did not automatically open; one atmospheric dump valve did not open; a letdown intermediate pressure control valve failed (causing the eves relief valve to lift); a rod bottom light did not light; a charging pump designated for emergency use only tripped 30 seconds after each of five starts; and, a pressurizer spray valve failed to resea As a result of the May 19 event, on May 21, 1986, the Region III staff directed the Palisades facility to shut down pending completion of an investigation into the cause of the May 19 reactor trip and permission of the Regional Administrator to restart following a briefing on corrective actions taken or planned by the licensee. Further details are provided in the confirmatory action letter dated May 21, 198 Accordingly, pursuant to NRC Manual Chapter 0514, Paragraph 042, this evaluation is necessar Ob~ective~_End_B~E~9.ns for the Backfit The objective of the backfit was to ensure that the causes and implications of the May 19 reactor trip, and the multiple equipment failures, including the burden these failures placed on the operators, were fully understood and corrected prior to the facility resuming power operation. Prior events at the facility, beginning in late 1984, due in part to inadequate mainten-ance, involved other problems with safety-related equipmen This included five events related to leaking Safety Injection Tank (SIT) check valves,

l

valve leakage problems on the HPCI injection line, SIT pressure control valves and a manual isolation valve, and the three-way divert valve in the chemical and volume control syste On March 9, 1986 the licensee elected to shut down and repair the problems but had to shut down again 16 days after returning to power operation after exceeding the Technical Specification limit for unidentified primary coolant system leakag Following the return to power operation on April 11, 1986, the licensee identified a packing failure on Condensate Pump 11A.

The pump was repacked twice prior to replacing it with an onsite spare. These events demonstrate a history of multiple equipment failures at the facility that are of concern to the NRC due to the potential for serious challenges to safety systems that they pose and due to the heavy reliance they place on continued above average operator response to maintain the plant in a safe operating condition. These concerns are supported by the final report of the NRC Region III Task Force Review of the Operational History (1983-1985) for Palisades, dated May 1, 1986 and the licensees SALP Category 3 ratings in the areas of maintenance, surveillance and quality program and administrative controls during the most recent SALP period ending October 31, 198 Safety Significance and A.2JlI2E.!'iaten~~~-2i Actj2~_Jak~~.

Appendix A to 10 CFR 50 sets forth principal design criteria for nuclear power plants which establish the necessary design, construction, testing, and performance requirements for structures, systems, and components important to safety that provide reasonable assurance that the facility can be operated without undue risk to the health and safety of the publi Included in these criteria are requirements to design systems which are capable of protecting the plant during anticipated operational occurrences with a single failure of an active system componen This single failure criterion is predicated on the assumption that the systems themselves are maintained in such a fashion so as to ensure a high degree of reliability, thereby decreasing the probability of multiple failures, a condition contrary to plant design base As noted above, the Palisades facility has had a history of poor maintenance and numerous component failure While it is recognized that not all of the recorded failures were with equipment important to safety, enough were to call into question the reliability of such equipmen The question of reliability of equipment important to safety is by itself safety significant; however, in the case of the Palisades facility, this significance is elevated by virtue of the numerous failures of equipment not explicitly important to safety. There are two reasons for thi First, failures of such equipment can and have caused unwarranted safety system challenges, increasing the frequency and complexity of anticipated operational occurrences. The net effect of this is that the probability of an accident is increased. This represents a direct adverse impact to safet The second reason is that increasing the complexity of an event places an unwarranted burden on the plant operator by requiring that operator to respond to multiple equipment failures with the attendant distraction

• that represents. The net effect is to potentially compromise the ability of the operator to respond in a fully appropriate and timely fashion to an event. This also represents a direct adverse impact on safet Because of the uncertain status of equipment at the Palisades facility and the number of unwarranted safety system challenges, the only viable option to ensure that no undue risk to public health and safety existed was to require the licensee to shut down the plant and evaluate its equipment statu Basis for Invoking the Exception In light of the multiple equipment failures that occurred on May 19, the licensee's demonstrated lack of conservatism regarding plant operations with deficient equipment and the licensee's SALP Category 3 ratings described above, the Region III staff determined that no alternative short of shutdown was feasible at this time because of the immediate need to ensure that this event and its implications were adequately understood and adequate corrective action taken or planne Persons Contacted F. W. Buckman, Vice President, Nuclear Operations J. F. Firlit, Plant General Manager R. M. Rice, Operations Manager R. D. Orosz, Engineering/Maintenance Manager D. W. Joos, Planning Coordinator R. D. Margol, QA Administr.ator R. E. McCaleb, QA Director T. T. Palmisano, Plant Projects Superintendent C. S. Kozup, Operations Superintendent R. E. Schrader, Electrical/I&C Engineering Supervisor C. M. Grady, Plant Mechanical Engineering Superintendent S. T. Wawro, Shift Supervisor J. F. Bowers, Senior Engineer, Mechanical Engineering G. S. List, ME&M W. T. O'Connell ME&M W. Clark, PE&C R. A. Fenech, Technical Engineer D. G. Malone, Senior Engineer, Licensing The AIT also held formal interviews and informal discussions with a number of licensee maintenance, operations, engineering and management personne.

NRC/Licensee Meeting On June 6, 1986 the NRC met with the licensee in Region III to discuss the progress and scoping of their corrective actions in response to the May 21, 1986 Confirmatory Action Letter. Additionally, communication interfaces between the NRC and the licensee were also discusse During

t * the meeting the licensee reviewed the May 19 event and outlined the progress of their independent review team efforts. Regarding the communication interfaces between the NRC and the licensee, both parties agreed that more concise, direct communication would be beneficia Conclusions The actions of the NRC Augmented Investigation Team were directed toward a review the circumstances surrounding the continuing equipment problems, as exemplified by the equipment failures which occurred during the May 19, 1986 reactor trip, and to examine the burden these failures have placed on the operating staff. The investigation proceeded simultaneously along two paths - equipment reviews and operations department interview The licensee had initiated a parallel effort in response to their commitments documented in the CA Although at the termination of the AIT onsite review, root cause determinations for several equipment failures were still in progress, the following significant facts were ascertained: Significant weaknesses exist in three aspects of the maintenance function - diagnostics, repair and post maintenance testin Plant operators have serious concerns regarding the adequacy of maintenance activities and equipment reliabilit Equipment failures and degraded equipment has placed varying levels of additional burden on the plant operator With regard to the May 19 trip this burden did distract operators, but did not significantly jeopardize plant safet The performance of plant operators and the operation of other major or safety-related plant systems were as expected and designed considering the equipment failure that occurre There is a lack of communication and coordination for a maintenance activity from the Work Request stage through verification testing and acceptance for operatio