IR 05000255/1989033

Insp Rept 50-255/89-33 on 891123-30,1206-09 & 15.Violation Noted.Major Areas inspected:891121 Inadvertent Depressurization Event Resulting in Reactor Trip & Safety Injection Actuation from Hot Standby

ML18054B382
Person / Time
Site:	Palisades
Issue date:	01/08/1990
From:	Axelson W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML18054B381	List: IR 05000255/1989033 ML18054B380
References
50-255-89-33, NUDOCS 9001240007
Download: ML18054B382 (27)

Text

U. S. NUCLEAR REGULATORY COMMISSION REGION II I Report No. 50-255/89033(DRP)

Docket No. 50-255 Licensee: Consumers Power Company 212 West Michigan Avenue Jackson, MI 49201 Facility Name:

Palisades Nuclear Generating Plant Inspection At:

Palisades Site, Covert, Michigan Wyle Laboratories, Norco, California License No. DPR-20 Inspection Conducted:

November 23-30 and December 6-9 and 15, 1989 Team Leader:

Inspectors:

Consultant:

Approved By:

B. L. Jorgensen J. K. Heller R. A. Westberg M. P. Huber T. G. Scarbrough R. M~~~ W. (. ft?e'i~~f Reactor Projects Branch 2 Inspection Summary 23-30 and December 6-9 and 15, 1989 (Re ort Areas Inseected: Special unannounced team inspection to review an inadvertent depressur1zation event on November 21, 1989, which resulted.in a reactor trip and safety injection actuation from hot standby. The causes appeared to be due to a single train of pressurizer power operated relief valve and motor operated block valve irregular operation. The Inspection Team evaluated: valve design, operation, and testing; procedures; operator actions and training; and, because the root causes of the event were not evident from inspection, a consultant was retained to observe dynamic (full flow) testing of the valve Results:

The inspection disclosed weaknesses in the licensee's integration of design and testing activities. A potentially serious event was precipitated by a valve's characteristic design response to certain test conditions, but the personnel conducting the test were completely unaware of this characteristi The root causes of the second opening of the pressurizer power operated relief valve (PORV) and*the PORV block valve failure to close_during the event could not be determined by either inspection or test. The licensee will meet with NRC Region III to discuss the corrective actions to prevent recurrence of this type event.

~

g

~'. :i. ;~* ~;g :~: '.~i?:

AUGMENTED INSPECTION TEAM REPORT U.S. NUCLEAR REGULATORY COMMISSION

REGION III

PALISADES NUCLEAR PLANT DEPRESSURIZATION REACTOR TRIP/SAFETY INJECTION FROM HOT STANDBY NOVEMBER 21, 1989 INSPECTION REPORT N /89033

. *

TABLE OF CONTENTS EVENT SUMMARY. * * * * * * * *. * * * * * * * * *. * * * * * * * *. * * * * * * * * *. * * * * * *. * * *

1 OPERATOR ACTIONS..............................................

3.. Training Factors Procedures Assessment PORV BLOCK VALVE M0-1042A *......... ~ *....**.*.**........*.....

Design/Application Testing Event Behavior/Anomalies Asse.ssment PORV PRV-10428 **. I ********************************************

Design/Application Testing Event Behavior/Anomalies Assessment *

SAFETY INJECTION ACTUATION....*......**..*....................

LESSONS L*EARNED * **.***..**..***. o *****************************

MANAGEMENT INTERVIEW (30703)......*......*...*...*.*.*....*...

ATTACHMENTS AIT Charter Sequence of Events Persons Contacted CPCO letter dated November 28, 1989

13

18

DETAILS Event Summary the Palisades Nuclear Plant completed an October/November 1989 maintenance and surveillance outage durin9 which the licensee replaced the Power Operated Relief Valves (PORVsJ and PORV block valves by removal of both the original pair of valves and welding in new valves. Target Rock supplied the relief valves and Rockwell-Edwards supplied the block valves, which they equipped with L imitorque motor operators. Post modification testing was to include an inspection of the welds in the section of piping between the block valves and their respective PORV This was scheduled to coordinate with the system hydrostatic test at the comp let ion of the outag On November 21, a primary coolant system (PCS) hydrostatic test* pressure of 2154 psi (at normal operating temperature) was obtaine Per the normal startup procedure, both block valves were closed, isolating their respective relief valves from PCS pressur Both PORVs were also close The Turbine Generator and Reactor Protection systems had been reset and tbe control rod shutdown banks were withdrawn for a pre-startup control rod test. At 2:25 a.m, the control rod test was suspended, and block valve M0-1042A was opened. *The associated PORV (PRV-10428) immediately opened also, starting a PCS depressurizatio The PORV stayed open for about 34 seconds and depressurized the PCS to around 1670 psi. This

caused a Reactor trip (due to Thermal Margin Low Pressure) and a Turbine Generator tri The PORV reseated: approximately 2 minutes later, it opened a second time for about 14 second The second opening caused an additional PCS depressurization to about 1565 psi and a safety injection signal (SIS)

resulte The operators reported several attempts to close the block valve during this time, but stated that they were not successful until after the PORV reseated for the second and last tim The plant was then stabilized in hot shutdow The licensee subsequently declared an Emergency Plan Unusual Event (UE)

at 3:05 a.m., because a PORV had been challenge The UE was secured at 3:34 The Licensee reported the event as required by 10 CFR 50.72 and a 10 CFR 50.73 written report is anticipate During conferen£e calls conducted on November 21 and 22, the licensee briefed NRC (Region III and Headquarters) personnel on the event and on potential courses of action. *The licensee stated that the unit would be taken to cold shutdowri to repair and trouble shoot the valves, but first, a safe means was needed to open a block valve without challenging the associated POR This would enable use of the PORV in its design function of establishing *the required low temperature overpressure protection (LTOP) during the cooldow ' *

Regi~n III established an Augmented Inspection Team (AIT), in coordination with the Office of Nuclear Reactor Regulation (NRR) to

~eview the events associated with the Palisades power operated relief and block valve irregularities. The. Team consisted of the SRI from D.C. Cook (Team Leader), RI from Palisades, and electrical and valve specialists from Region III and NR The Team Charter (Attachment 1) included:

validation of the events sequence; determination of apparent equipment irregularities; assessment*

of recent valve design changes and testing; and, review of operator training and performanc Concerning the first item of the Charter, a detailed, annotated sequence of events is included as Attachment 2 to this repor Some Team members began chartered activities on November 23, with the entire complement on site by November 2 Persons contacted in the course of Team activities onsite are listed on Attachment The first order of business was monitoring licensee actions to safely achieve cold shutdow To enable the plant to be taken from hot to cold shutdown, the lfcensee generated a special operating procedure and a temporary modification to the block valve control circuitry to allow opening a block valve while minimizing the probability and impact of another inadvertent PORV openin The Inspection Team reviewed: Temporary Modifications (TM)

No.89-087 and No.89-088; Safety Review No. SE-791-02, Revision B; Work Request No. 140706; and Drawing N E-~42 Sheet 4, "Schematic Diagram Volume Control Shutdown Cooling & Pressure Relief Motor Operated Valves,"

Revision 16 (see Charter, Item 5).

The Team also discussed the event and operation of the block valves with cognizant plant electrical majntenance personne Review of TM 89-087, TM 89-088,'and Drawing No. E-242, verified that removing the seal-in feature on the block valve opening *cycle would give the control room operator better control over the block valv This would allow the operator to stop the valve as it opens and immediately reclose it if a problem develope The operator could a 1 so "bump" the valve off its seat to allow a more gradua 1.

pressurization of the pipe between the block valve and the PORV; and to reduce the probability of inadvertent PORV actuatio Review of Safety Review SE-791-02, indicated that the temporary modification to the block valves did not involve an unreviewed safety questio Inspection Team members attended the Plant Review Committee meeting on the temporary modification and special procedure on November 24, 1989, and observed implementation of these activities on November 2 Th~

actions were successful; PRV-10438 was unisolated and placed in service for LTOP and the plant safely achieved cold shutdown on November 26, 198 ' * Operator Actions At the time of the event, approximately 2:25 a.m. on November 21, 1989, the Palisades Control Room was staffed with two licensed Reactor Operators; designated Control Operator No. 1 (C0-1) and Control Operator No. 2 (C0-2)

by the licensee. Present in the Shift Supervisor's office were the Shift Supervisor (SS) and the Shift Engineer (SE), both holders of Senior Reactor Operator licenses. A normal complement of auxiliary operators was onsite; they were responding to an unannounced fire drill in the condensate pump area and we*re not materially involved in the

-

depressurfzation event and its immediate aftermat The C0-2 was performing a routine startup test of the control rod interlocks. Several control rods had been tested when the request was received (through.the SS and the C0-1) to open valve M0-1042A, one of the Power Operated Rel}ef Valve (PORV) block valves. This valve had previously been closed (along with the other motor operated PORV block valve, M0-1043A) as specified by the startup procedur The change in valve position required an administrative control device, called a special valve lineup, to ascertain the acceptability of changing the valve to an off-normal positio The SS verified the special valve lineup sheet included the opening of the PORV block valves; other valves had also been repositione The following d1scusses the observations and actions of the operators, as*

described in subsequent interviews and statements prepared for the post trip review (see AIT Charter, Item 6). These interviews yielded quite consistent descriptions for all events witnessed by more than one perso Where there'was only one observer, the narrative so indicate The C0-2 turned the handswitch for valve M0-1042A to "OPEN".

Within*

a couple of seconds, as expected, the red "OPEN" indicating light above the handswitch illuminated, so*both the red and green (CLQSED)

lights were lit. Almost simultaneously, the associated PORV (valve PRV-10428) opened as wel PORV open alarms were received and primary coo.lant system (PCS) pressure began a very rapid drop, as shown on a front-panel LED readou PORV design and behavior are discussed further in another section of this repor One or both control operators called out that the PORV was open, which caused the SS and SE to enter the control roo The C0-2 reversed the block valve handswitch to the CLOSED position and held it there (it would otherwise spring-return to neutral) but he knew the valve had a "seal in" circuit and had to stroke fully open before it could be closed. A full stroke consumes approximately 18 second He believed he saw the block valve reach OPEN (red light only) then return to red/green, indicating it had begun to clos He stated he continued to hold the handswitch to "CLOSE", for what seemed an equivalent time a*s the OPEN strok He then released the switch (still showing red/green) and checked various plant parameter Glancing back at the block valve, he observed it OPEN (red light only), so he gave it another CLOSE signal and observed concurrent red/green light By this time, a reactor trip had occurred, and the C0-2 left the block valve handswitch to perform associated immediate actions".

The C0-1 was likewise performing his "immediate actions". Only the C0-2 indicated he saw the block valve OPEN twice. The C0-1, the SS and the SE all stated they did not see the valve fully ope The design of the valve would not permit closure without it first being fully open once, barring some electrical proble No such problem could be subsequently identified in multiple tests performed by the licensee, some of which were witnessed by a member of the Inspection Tea Therefore, it appears likely that the valve did fully open at least onc It is less certain that the block valve.was fully open only onc The data logger shows a continuous "NOT CLOSED".entry for*

nearly three-and-one-half minute The timing, the design and the post-event checkouts seem to exclude the possibility the valve moved in the OPEN direction other than in the initial, approximately 18 second interval. The reactor trip occurred at 24 seconds after the block valve first indicated "NOT CLOSED", around 26 seconds after the initial OPEN signal from the handswitc This is insufficient time for this valve to come fully open (about 18 seconds), then close for an approximately equivalent time (another 18 seconds); be unobserved briefly, found OPEN again (an6ther 18 seconds) and rein~t~ucted to CLOS That would require around 60 second At the time the C0-2 left the block valve, it was indicating *

red/green concurrently. Sufficient time had passed (at least 30 seconds) to provide confidence that if the valve was moving, it was moving in the CLOSED direction. This is supported by repeated subsequent observations that the valve always showed red/green until it finally showed green (closed) onl When opened, the block valve moves until stopped by a limit switc When closed, the valve moves until stopped by an overtorque condition, presumably from the valve disc driving into the sea Any other cause of indicated overtorque (friction, obstruction, torque switch maladjustment) would cause a closing valve to stop in place, where it would remain until the torque switch cleared and the handswitch was operated agai Additional details of 5Tock valve design and operation are discussed in Paragraph The reactor trip "immediate actions 11 are memorized checks performed by both the C0-1 and the C0- These checks consume about a minute or so, barring complications, and involve movement around and behind some of the control panel During this process on November 21, or perhaps slightly before it began, the PORV reclosed at about 34 seconds into the event. All control rods had tripped as designe No significant abnormalities were eviden Both operators stated the C0-1 first identified the PORV had close PCS pressure stabilized and b~gan to return toward a "normal" pressure above 2000 psig, from approximately 1670 psi The more formal, routine post-trip checkoff was underway when the PORV opened agai Neither operator recalled operating any controls coincident with this even The second PORV-open cycle was quicker, lasting only about 14 seconds, but reducing PCS pressure belo\\'1 the 1605 psi setpoint for safety

• injection actuation signal {SIS).

The SIS immediately sequenced and loaded all required safety equipment onto its respective emergency bu This depressed bus voltage and resulted in both emergency diesels starting; however, offsite power remained in service so neither diesel loaded to its bus *. The effect of SIS on the emergency electrical systems is discussed in greater detail in Paragraph Another CLOSE signal was given to the block valv This time the C0-1 operated the control and the valve finally closed about 26 seconds (more than a normal full-stroke time) after the PORV had closed. This ended the even The SS subsequently chose EOP-4.0, "Loss of Coolant Accident Recovery"-, to verify plant status and stabilize in a safe conditio The Inspection Team considered this a prudent, correct choice. *

In addition to conducting interviews w1th the operating shift crew, the Inspection Team reviewed applicable procedures and instructions, logs, and training records (see Charter, Item 4) in order to assess operator performanc *

a~

Training Factors Two types of training, germane to the event, were provided to the involved shift cre Classroom training specific to the PORV and block valve modification was accomplished under Lesson Plan IE 89030 Facility Changes FC-791 (rep 1 a cement of PO RVs a_nd b 1 ock valves with new) and FC-809, (new PORV actuation and control circuits) were each described. System functions, norma 1 operation and failure consequences were discussed. *An exercise challenged the trainees to determine Low Temperature Overpressure Protection (LTOP)

system operability requirements. The classroom training emphasized the LTOP function of the syste The Inspection Team found no classroom training involving either operation *of the plant with the block valve open, or opening the block valve while at high pressur The second type of training consisted of review of proposed revisions to procedures, an activity accomplished by a "Required Reading" assignment as on-the-job training (OJT).

The procedures affected by the new PORV and block valve were reviewed by the Inspection Tea None contemplated changes to the operating mode previo~sly practiced; namely, the block valves were to be closed as part of disarming LTOP during heatu Procedures Plant procedures relating to normal startup and operation, and to arming and disarming the LTOP system, (Charter, Item 3) were reviewed by the Inspection Tea At the completion of the outage, the licensee began an orderly plant heatup to hot shutdown using General Operating Procedure (GOP)- Steps 2.29 and 2.41 established the conditions that subsequently led to the inadvertent opening of the POR *

Step 2.29 disarmed the LTOP system by reference to System Operating Procedure (SOP)-1, "Primary Coolant System.

Section 7.1.4.0, of SOP 1, closed the two block valves (M0-1042A & M0-104-3A), placed HS-105A & HS-1058 in defeat and placed the PORVs (PRV-10428 &

PRV-10438) to the closed position. This was done when the cold leg temperature was between 385 degrees Fahrenheit and 430 degrees Fahrenheit. Thus, normal plant procedures isolated the PORVs before significant plant pressurization as part of 11disarming 11 the LTOP syste Step 2.41 of GOP 2 required performance of a PCS leak test (if the PCS was opened) per R0-70F or SOP 1, Attachment The Operations Administrative Support Supervisor (an individual holding an active SRO license) chose SOP 1, Attachment 2, which established a test pressure of 2150 psi and a temperature of 532 degrees Fahrenheit or greate The attachment does not normally require valve manipulatio However, Temporary Change Notice (TCN-0-89-286) was made to Attachment 2 and established a special valve lineup for VT-2 examinations during the PCS Hydrostatic test. The lineup was written by the Operations Administrative Support Supervisor and contained the valve manipulations necessary to expose the PORV modification (FC-791)

to hydrostatic pressur The lineup was written by reviewing the Repair/Weld Inspection Cards (RIC) for the applicable work order (W0-24903975), which implemented FC-79 The RIC specified a hydrostatic test pressure of 2135 psi and a temperature of 500 degrees Fahrenheit or greater. The 11C 11 shift (4:00 p.m. to Midnight) Operations crew performed the special valve lineup on November 20, 1989, except for~

manipulation of the two PORV blo~k valves. The following 11A shift crew subsequently opened one block valve (M0-1042A) and precipitated the event of interest in this repor The Inspection Team reviewed_ the valve lineup implemented by TCN-0-89-28 Plant Administrative Procedure 4.02, "Control of Equipment Status" at paragraph 8.0.b, established the requirem~nts for special valve lineup sheets which alter an existing mode of operation or which manipulate-systems in a manner not covered by an SO Paragraph 8~0.b required: Shift Supervisor approval of the lineup; that the lineup be implemented by a Temporary Change Notice; that the Temporary Change Notice not become permanent; and, system restoration be specified on the specia 1 1 ineup shee The. Temporary Change Notice and the special valve lineup complied with these administrative requirement As discussed previously however, the position of the PORV block valves is controlled by paragraph 7.1. of SOP 1 and step 2.29 of GOP 2,* prior to full heatup and pressurization. Manipulation of the block valves, thereafter, alters the purpose and scope of those proc_edure This means manipulation of the valves by a special valve lineup and Temporary Change Notice was not the correct mechanis Manipulation of the PORV block valves, after heatup, involved a procedure revision, which required a safety evaluation pursuant to 10CFR50.5 This is discussed further below *

6 Assessment The Palisades Plant operators were-not trained for the condition of operating the plant with the PORV block valves open, nor were they trained concerning.the potential consequences 'of opening the block valves at high PCS pressur Once the event began, reported and recorded operator actions were apparently without error and were pruden The Palisades Plant procedures intended for the PORV block valves to be closed prior to significant PCS pressurization as part of disarming LTOP and did not address plant operation thereafter with the block valves ope No reviewed, approved procedure existed for opening the block valve at high PCS pressur Opening the PORV block valve*M0-1042A at a high PCS pressure constituted a change to the intent of a plant procedure (SOP-1)

which was reviewed and approved pursuant to Technical Specification 6.8.1.a. As such,-accomplishing this action as a

"temporary change" is contrary to Technical Specification 6.8.3.a, which permits only such temporary changes as do not affect the intent of the original procedure; this appeared to be a Violatio The NRC resident inspector office was assigned to further review and to iriitiate any appropriate regu_latory action on this matte The Inspection Team did not consider this apparent violatiori as a root cause of the even Further, it was not obvious that processing the required procedure revision (or preparing a special procedure)

• and performing a 10 CFR 50.59 safety evaluation would have necessarily prevented the even One potential violation and no deviations, unresolved or open items were identifie.

PORV Block Valve M0-1042A Design/Application The Inspection Team reviewed design change documentation (see Charter, Item 7) associated with the newly installed block valve The review included portions of Facility Change (FC) No. 791, "Pressurizer Valves Replacement Project," Revision 2, Specification N SP-MP-8304-00l(Q), "Replacement Power Operated Relief Valves,"

Revision 0 and No. SP-MP-8304-002(Q), "Replacement Power-Operated Relief Valves," ~evision O; Report No. MP12-1141, "Third. Party Qualification Review of Replacement valves *For Palisades Nuclear Power Plant," dated August 27, 1989; Technical Report No. TR-7258-1,

"Palisades Pressurizer Valve* Project Third-Party Qualification Review," dated August 24, 1989; and Safety Review No. SE-791-01

"Palisades Pressurizer Valves Replacement Project, 11 Revisions C, D, and *

(1)

The team concluded that FC-791 and the Safety Evaluation of the conceptual design were acceptable; however, in general both only considered the use of the PORVs and the block valves for use during Once-Through Cooling (OTC) and Low Temperature Over Pressure (LTOP) operation~

_

(2)

Hydro testing of the modified relief system from the pressurizer to the PORVs *was required by paragraph 4.1.17.3 of the Design Plan; however, the Safety Evaluation did not address that this test would be performed after heat up {2154 psig). Opening the block valves after achieving Hot Standby is an action not described by the plant's procedures, as discussed above_

(Paragraph 2.c), and could be classified a "special test".

• Failure to perform a Safety Evaluation prior to performing a special test is a potential violation of 10 CFR 50.59. This potential Violation is, in effect, another way of approaching the issue of review/approval of an unusual plant configuration identified as a potential Violation in Paragraph 2.c above. As noted above, the NRC resident inspection office will evaluate the matter further and make a specific recommendation concerning any enforcement action deemed appropriat (3)

The specifications for the PORVs and the block valves were acceptable; however, they too were concerned mainly with their operation during OTC and LTOP, even though the block valve specification required the block valve to be qualified to close against a differential pressure of 2500 psi (4)

The unique operating characteristics of the Target Rock Valves when being subjected to high differential pressures (discussed in Paragraph 4.c, below) were not considered in the FC-791 desig *

In the Palisades design, two PORVs (PRV-10428 and 1043~) are attached to the pressurizer with each relief line having a motor-operated block valve (M0-1042A and 1043A).

During power operations, th~-block valves are closed, as stated in the FSA The PORVs are used during shutdown conditions for low temperature overpressure protectio The original lines and valves were too sma 11 for a single line to accommodate post-accident "feed and bleed." This method of core cooling relies on the safety injection pumps to supply water to the reactor coolant system, while core heat is removed by rejecting water from the relief line To enable each relief line to have adequate capability, the licensee modified the lines by increasing their size from 2.5 to 4 inches and by installing larger PORVs and block valves. This modification was in response to Item II.D.1 of NUREG-073 The modification also improves the low temperature overpressure protection capacity of the relief line In designing the modificatiori, the licensee provided the design specifications for the block valves to Ed\\'.i'ard Valve (a subsidiary at

that time of Rockwell International).

The design specifications included, among other things, differential pressure conditions, system pressure and temperature conditions, valve opening and closing times, and environmental conditions. Edward Valve selected each PORV block valve to be an Edward 4 inch "Equiwedge" type gate valve with a Limitorque SMB-00-25 motor operato Based on the licensee's design specifications, Edward Valve calculated and set the torque switch settings before delivery to the license The licensee reviewed the sizing calculations as provided by Edward Valv The licensee indicated that this procedure is typical for its purchase of MOV * Testing Testing of the block valves was reviewed by the Inspection team (see Charter, Item 3) including vendor test data. The block valves were tested by the supplier for opening against fu 11 differentia 1 pressur The valves were reportedly not tested for closure under full flow conditions because of the absence of a suitable test facilit They were considered capable based on a "reference" test of a three inch valve, along with calculational modelin Following installation, the licensee performed post-maintenance testing of the block valves under no load condition During testing of M0-1043A for its open stroke, the motor was burned out when the limit switch failed to trip the moto The cause was traced to the failure of the supplier to install a "pin" in the assembl This was corrected and the motor was replace Block valve M0-1042A was inspected and found to contain the required pi The Licensee had a Valve Operator Test and Evaluation System (VOTES)

diagnostic analysis performed_ on each MOV to verify that the torque switches tripped at the expected thrust values. This VOTES testing was performed at a zero differential pressure conditio The VOTES testing also provided additional information such as indication of**

the accuracy of packing load assumptions~ Although not previously

~he case, the current block ~alves are classified as ASME Section III, Class 1 and are included in the inservice testing program for Section XI of the ASME cod With respect to the licensee's maintenance progr,am for MOVs, the licensee had a list of the required thrust values for each safety-related MO Following a maintenance activity that might affect MOV operation, the licensee has a VOTES diagnostic analysis performed to determine that the MDV is delivering the expected thrus Verifying thrust empirically means the licensee relies very*

little on the value of the torque dial indicator settin On the other hand, reduction in available thrust margin could result from changes in packing load, and this needs to be monitore This was discussed with licensee representative Event Behavior/Anomalies Following the modification to install the larger PORVs, block*

valves, and piping; the licensee attempted to perform a hydrostatic test of the new installatio As part of this test, the licensee opened* block valve M0-1042A at 2:25 a.m., on November 21, 1989, with pressure at 2154 ps Immediately after initiating the opening of M0-1042A, the PORV in that line (PRV-10428) opened and began sending steam to the quench tan Upon recognizing that the PORV was open, the control room operator turned the block valve hand switch to th~

close position. The block valves, however, had a seal-in circuit that required the valve to stroke fully. Therefore, the control room operat6r could not begin closing the MOV until it had opened full The computer printout data indicate that the MOV remained "not closed from 02:25:50 to.about 02~29:18. Because the data are obtained from a full-stroke switch, the actual valve disk position is not know During that three minute and 28 second interval, the pressurizer pressure dropped rapidly for about 34 seconds (while the PORV was also open), began to slowly recover for two minutes and 10 seconds, then dropped rapidly for 14 seconds when the PORV opened again, and finally began to recove Full closure of the MDV was not indicated until 26 seconds after the pressurizer pressure began to recover after the final PORV reclosur The licensee's documentation indicated that the block valve operated properly during the VOTES diagnostic testing with no differential pressur The torque switch tripped at the expected valu Therefore, if the thrust calculations were correct, the control room operator should not have required three minutes and 28 seconds to allow the MOV to open fully and then shut the valv The valve stroke time, during the VOTES testing, was less than 20 second The AIT reviewer examined thrust calculations provided by the valve supplie Based on the input parameters given, the expected thrust requirements were independently recalculate The calculations were found mathematically correc The motor and actuator sizes met. the calculated thrust requirement The VOTES test documentation indicated that the torque switch settings were consistent with the required thrust calculation at about 12,000 pound One apparent weakness in the licensee's calculations was that the packing load was derived by the valve supplier. That value was 685 pound The VOTES testing, however, indicates that the actual value was about 300 pounds higher, close to 985 pound This under estimation of packing load slightly reduces the available thrust margin~ but this did not appear to be significant to the subject even The documentation did not indicate that the MOV was evaluated for accommodating stall thrust on the maximum valve thrust ratin If the MOV was operating as designed and installed, but nevertheless failed to close under design conditions, then an appropriate -

J assumption is that the input parameters into the thrust calculations were incorrect. System pressure, differential pressure, and seat and st~m.areas were all found to be correct in the calculation Therefore, a primary candidate for an incorrect value was the valve friction facto The valve supplter calculated this factor based on sliding friction tests. A normal distribution of friction factors was found and a value of 0.37 selected that predicted that 99 percent of all friction factors would be below this value. It is not clear that this statistical approach is conservativ Further, simple sliding tests are not representative of friction factors where closure is against high flow with a moment exerted on the valve seat. Research*and operating experience indicate that a valve friction factor of 0.37 may be only about half of the actual value for similar types of gate valves.,

An unexpectedly large valve friction factor could result in the torque switch being set too *1o For example, increasing the friction factor for this valve to 0.50 would raise.the required thrust from about 12,000 pounds to almost 15,000 pound Because the SMB-00.actuator has a maximum thrust rating of 14,000 pounds, the adequacy of the actuator would need to be confirmed if the torque switch settings.were.to be increase Assessment After the event, the licensee disassembled the block valve to obtain measurements of the valve and to inspect for conditions that might have caused the inadequate valve performance, such as stem or disc galling, metal-to-metal transfer, or unusual wear pattern No such conditions were noted that.would provide positive proof of a proble Valve measurements were obtained by the licensee to verify that the tolerances were acceptabl Stroke testing at static conditions was performed following the even The block valve was instrumented using the VOTES equipment to evaluate the thrust developed by the operator. This stroke testing did not reveal any problems with the valv The thrust at torque switch trip (the limit that would halt the valve motion) in the closed direction was 12,363 pound This value was greater than the value estab"lished by the vendor to ensure valve. closure, and was consistent with the*pre-event settin The AIT Team was unable to absolutely determine the root cause for block valve behavior (see Charter, Item 2) from information available at the plan To evaluate the operability of the valves, the licensee removed the PORV and the block valve and shipped them to the Wyle labo~atories, Norco Facility, for testing. The licensee arranged to have assistance in conducting the tests from Edward Valve, Target Rock, Bechtel/KWU, and Babcock & Wilco The tests*

were observed for the NRC staff by its consultant EG&G, Idaho, which has expertise in the testing of valves under high differential *

pressure condition The physical arrangement of the PORV and its block valve at the test facility was said to be similar to the actual plant configuratio Because of its small steam supply volume, the test facility could not maintain full system pressure throughout the block valve closing strok As will be discussed later, this resulted in an inability of the test facility to fully model the November 21 even At Wyle, the licensee performed tests of the block valve for its capability to open or close against various differential pressure condition Many of the tests were run to allow independent data collection. by Liberty Technology and B&W diagnostic system Among the data collected, thrust measurements were taken by means of a VOTES sensor on the valve yok During the testing of the PORV on December 6, the block valve was opened electrically against 575 psig on several occasion On December 7, the MOV was opened against differential pressures of 1760 psig and 2500 psi The block valve was closed against differential pressures of 43 psig and 700 psig. A successful seat leakage test of the block valve was performed with system pressure at 2500 psi On December 8, the licensee performed four tests aimed at demonstrating the-operability of the block valve under differential pressure conditions. Because of the limited volume of the steam supply, the test apparatus was unable to supply full system pressure during the entire close stroke of the block valve. Therefore, the licensee performed closure tests.of the block valve from partial open positions. For each of the four tests, the.initial system pressure was 2500 psia but dropped to lower values during the closing strok In the first test, the block valve was initially positioned at 60% open when the PORV was opened to commence blowdow The block valve was then given a signal to clos The block valve successfully close The final system pressure for that test was 2000 psi The block valve was jogged open against this pressur In the second test~ the block valve was again initially positioned at 60% open wh~n the blowdown bega The block valve successfully close The final system pressure for that test was 2000 psi The block valve was jogged open against this pressur In the-third test, the block valve was initially positioned at 25%

open when the blowdown was begu The block valve successfully close The final system pressure for that test was 2430 psi The block valve was also jogged open against this pressur In the last test; the block valve was initially positioned at 100% open when the blowdown Was begu The block valve successfully close The final system pressure for that test was 1730 psi Because the test facility could not fully duplicate plant conditions, detailed analysis of the results of these tests including the licensee's instrumentation data will be necessary to determine whether the tests were adequate to demonstrate that the block valve

'. ' -~.

will operate under co~ditions similar to the November 21 even Further, the reliability of closure tests from partially open positions in demonstrating MOV operability from a full open position will need to be evaluate In this regard, the nonlinear increase in tequired thrust d~ring valve closure that has been found in ce~tain valves as part of NRG-sponsored research of high flow conditions may be relevan One potential Violation, and no deviations, unresolved or open items were identifie.

PORV PRV-10428 Design/Application The Inspection Team reviewed design change documentation (see Charter, Item 7) associated with the newly installed Power Operated Relief Valve The Palisades PORV is a Target Rock brand two-stage pilot operated relief valv It is solenoid actuated to allow for the control of either steam or water flo When the valve is closed, inlet pressure is transmitted through control orifices in the main piston to a control chamber above the main piston and dis Pressurization of the control chamber results in balanced forces on the piston and _leaves only the force created,_on the main disc by the inlet pressure; this provides-a strong seating force. Additionally, return springs on the pilot and main discs provide an additional seating forc '

When the solenoid is energized, a magnetic force is developed and the pilot disc is opene This closes the control orifices and allows the pressure in the control volume above the piston to vent through the pilot seat to the downstream side of the valv The resultant loss of control pressure allows the inlet pressure acting on the main piston to move the disc to the open position. The main disc is then held firmly in the open position by a larger magnetic force created by coupling of the upper and lower plunger At the top of the valve, a reed switch assembly provides indication of valve position and input to the plant events recorde The input to the events recorder indicates only that the valve is not close The reed switches are actuated by the motion of a magnet assembly through the upper plunge PORV closure is accomplished by de-energizing the solenoi When the solenoid is de-energized, the magnetic opening force is eliminated and the pilot return spring closes the pilot valve, reopening the control orifices. The control pressure then increases

• until, in combination with the main spring, closing force exceeds the pressure acting on the annular area of the main disc, allowing the PORV to clos Testing Testing of the PORV was reviewed by the Inspection Team (see Charter, Item 3) including procedure number T-FC791-8304-501Q, Revision 0, "Power Operated Relief Valves." This procedure was approved and released for implementation on October 2, 198 The-testing verified that the relief valve was properly installed and would operate. Tfie stroke times for the PORV were measured to be 0.7 seconds (open) and 0.42 seconds (close). These values for stroke time were less than the design values of less than 2 second Testing was also conducted to demonstrate operability and to test fo~ degradation of the PORV by functionally checking the PORV through the actuation channel of the LTOP system and verifying valve full open stroke and open stroke tim Response time testing was also performe The stroke times were taken with the inlet pressurized from 175 to 225 psig and were found to be 1.275 seconds, less than the design value of less than 2 second No problems with these procedures were note Event Behavior/Anomalies During the event, the first opening of the PORV exhjbited a characteristic of valve desig The key to valve operation is that the inlet pressure acts on both the top and under sides of the piston (balanced opening and closing forces) and on top of the disc (closing force). Since the piston has a larger cross-sectional area than the disc, opening forc_e will be greater than closing force whenever pressure is acting only on the under side of the piston. A valve of this design would a~open, however, if a large and rapid enough transient inlet-pressure were applied, sufficient to overcome disc/stem weight and closing spring force Therefore, a pressure surge capable of creating a temporary pressure difference of 85 psid, between the inlet area and the control volume (connected through an orifice), would be large enough to open the PORV *

. The PORV handswitch ~as in the "Clos~" position and there was no electrical sfgnal to open the PORV during the event. Therefore, the pilot disc was seate With a closed pilot disc, the control orifices were open and inlet area and control volume pressures _

(above and below _the-pis.ton) were very low and equalize When the block valve was opened, steam at approximately 2150 psig very rapidly pressurized the small (approximately one-half cubic foot)

inlet.area volum Pressure buildup fiom the opening of the four inch valve substantially exceeded equalization through the orifices (two bores at 0.060 inches each) to the control volum Thus, the

valve disc lifted. The lift was subsequently computer modeled and found to have been sufficiently rapid that the pilot valve, internal to the main disc, may have been thrown open against its closing spring. This could have delayed pressure buildup in the control volum The primary factor in the 34 second duration of the initial PORV opening, however, was found to be condensation in the control.

chambe Testing at the Wyle Laboratories Norco facility showed unpowered PORV opening and closing cycles from a few seconds to over 60 seconds, depending.on temperature differences among the steam supply, valve inlet, control chamber and outlet. With the PORV mounted vertically, condensation in the control volume apparently drains out of the small orifices very slowly, if at all, such that steam subsequently forced through the orifi~es is "quenched" and control volume pressure buildup is delayed until the condensate boils off. Computer modeling which incorporated this phenomenon calculated valve opening durations consistent with those observe Assessment The team observed the post-event disassembly and inspection of the PORV which the licensee performed to provide information to aid in the root cause analysi During the disassembly of the PORV, foreign material was found in the valve that was similar to wood shavings. Additionally, the main disc piston rings were found to have their plating to be flaking off. These conditions, though abnormal, were not believed likely to cause the PORV to operate erraticall No other problems were noted at the time of the inspectio The reed switches on the PORV were verified to be in the correct positions to provide valve position indicatio PORV full stroke was also verified during the inspectio Subsequent testing of the PORV, in tandem with block valve M0-1042A, was performed by the licensee at the Wyle Laboratories Norco, CA. facilit On December 6, 1989, the licensee performed tests on the PORV at 575 psi The purpose of the test was to open and close the PORV electrically at this system pressur Both of the tests conducted in this manner were successful. Another type of test was intended to determine if the PORV would pop open following the*

opening of the block valv For four such tests, the PORV and its block valve were initially close The block valve was then jogged open for 2 to 3 second In each case, the PORV popped open and remained open fo~ intervals of 2-4 seconds, 21 seconds, 64 seconds, and 14 seconds, respectively. After jogging the MDV open and waiting for the PORV to reseat, the MOV was opened full The PORV did not *

spuriously reopen in any of those four tests during the full opening stroke of the block valve.

• On December 7, 1989, PORV tests were performed with increased system pressure. First, with the system at 1760 psig and both valves closed, the block valve was jogged ope The PORV popped open briefly and reseate The block valve was then opened to 50%,

closed, and fully opene The PdRV did not pop open during this stroking of the block valv The licensee then opened the PORV an block valve to blow down the syste Following the blowdown, the licensee closed the MOV but, unknown to the licensee, the PORV apparently failed to fully close. With the system pressure at 2500 psia, the block valve was opene Pressure fell rapidly through the open POR At 700 psig, the licensee closed the block valv Upon removing the PORV, the licensee found. a 0.75-inch metal plate lodged in the POR The licensee will need to inspect the PORV and block valve to ensure that the metal plate did not damage the valve During the testing of the block valve on December 8, the ~lock valve was jogged open on three occasions with the system pressure at approximately 2200, 2000,* and 2430 psig, respectivel In each case, the PORV popped open and then reseated in 3 to 5 second Through the testing at the Wyle facility, the licensee confirmed that the PORV will pop open if it is subjected to a sudden surge of steam that results when the.block valve is opene The length of the interval during which the PORV remained open ranged from 2 to 64 second This extensive range indicates that the block valve must be capable of closing during full flow through the POR The licensee was unable to duplicate the event at Palisades where the PORV. appeared to pop open a ~econd time during the transien It is not known whether the small steam volume of the test facility was a factor in the inability to cause the PORV to pop open a. second tim In any event, the licensee should review the event data to confirm the initial conclusion that the PORV did pop open the second tim If that conclusion is confirmed, the licensee should ensure that the PORV circuitry is correct and that a spurious open signal was not sent to the POR The tests focused on the November 21 event.in that steam was the only fluid used during the tests. Therefore, the tests might not be useful in satisfying the requirements of Item II.D.1 of NUREG-073 For example, the PORV might need to operate with saturated water condition The AI Team concluded that the subject event was precipitated by a predictable* respo~se of the Target Rock brand PORV to a sudden inlet pressure increase. Other brands of pilot actuated PORVs have similar design principles and would respond similarl It is Linc.lear who may have known about this characteristic of PORV behavior. Those closely involved in setting up and conducting the

hydrostatic test did not kno Had they known, they would not have set up the test in the way they did, and the event would not have occurre On the recommendation of th~ valve vendor, the Palisades PORV is being reinstalled rotated some 135-degrees from vertical. - This is.

intended to minimize the collection of condensation in the control volum Depending on individual valve internal geometry, it is considered probable that pilot actuated valves from other vendors could also be susceptible to prolonged openings in unpowered actuations. *

The second PORV opening remained unexplaine The details of design development, of procurement, *of testing, a*nd of procedures and training, all emphasize the new PORVs as a low temperature and low pressure functioning syste The breakdowns in communication among various groups having an involvement in the installation of the new system were not all readily apparent, but several opportunities existed to change the course of event The licensee decided to investigate these factors and opportunities via the Human Performance Evaluation System {HPES) process developed by the Institute for Nuclear Power Operations {INPO).

Following the HPES evaluation is beyond the scope of the AIT Charter, but the results (findings and corrective actions) will be of interest to NRC Region III, and perhaps other *No violations, deviations, unresolved or open items were identifie.

Safety Injection Actuation Charter Item No. 8 concerns the circumstances surrounding the safety injection and the loading of the safeguards bus, which caused the undervoltage condition and the diesel generator start. The Inspection Team reviewed the event sequence, portions of Faci1 ity Change (FC) N,

110ffsite Power Reliability Modification," Drawing No. E-1, "Plant_

Single Line Diagram, 11 and Logic Diagram No. E-17, 112400 Volt Load Shed.

This review produced the following information: Two minutes and 58 seconds into the event, PCS pressure fell to 1605 psia which actuated the Safety Injection System {SIS). The large inrush of starting current, due to the actuation of multiple pumps an_d valves by the SIS, dropped the voltage on the 2400 volt safeguards buses IC and ID to.approximately 87 percent of norma The degraded voltage protection relays on the buses started Diesel Generators (DGs) 1-1 and 1-2, since the.bus voltage reached the set point of voltage less then 92 percent for greater than one half secon d~

As the SIS pump motors accelerated, the starting currents decreased, and the bus voltages returned to greater than 95 percent in approximately four seconds. This was less than the load shed setpoint of voltage less than 92 percent for greater than six seconds; therefore, these buses remained on off-site p,ower and the diesels were not loade The Inspection Team determined that the SIS, diesels, protection logic, and load shed logic performed as designe No violations, deviations, unresolved or open items were identifie.

Lessons Learned A significant adverse event occurred at the Palisades plant which was precipitated by a predictable response of a pilot-actuated relief valve to a sudden inlet pressure increase. This valve design characteristic, however, was not generally appreciated, and was certainly unknown to the operations and test personnel directly involved in setting up the test conditions which caused the proble The test conditions, though unique and not specifically addressed in design documentation, training or procedures, were established routinel The event thus emphasized the need for thorough evaluation, understanding, communication and coordination before performing any plant manipulation for the first time, particularly those involving new or modified system This event emphasizes an NRC concern for the ability of MOVs at other plants, not specifically-tested under high differential pressure and flow conditions, to operate under those conditions. Although this licensee historically operated with the block valves closed, many plants operate with the valves ope Such plants may need to reexamine the bases for believing that their block valves will operate under design basis condition Events have occurred at other plants where MOVs failed to open or close against high differential pressure and flo The Palisades event is additionally significant in that it resulted in an unisolable

• (until the PORV closed) loss of coolant event. This event again highlights the need for licensees to implement a program to ensure MOV operability under postulated pressure/flow conditions discussed in Generic Letter 89-10, "Safety-Re lated Motor-Operated Valve Testing and Surveillance.

This licensee completed the Bu 11 et in 85-03 program and performed *fu 11 differential pressure testing of about 20 of the 24 MOVs within that progra Another 20 MOVs were said to be within the scope of Generic Letter 89-10 for Pali s*ades. Other plants may not have tested their MO Vs at full differential pressure to such an extent as this license ** Management Interview (30703)

The Inspection Team met with licensee representatives on November 30, 1989, and again on December 15, 1989, to discuss the scope and findings of the inspectio In addition, the inspector also discussed the likely informationa 1 content of the inspec.tion report with regard to documents or processes reviewed by the inspector during the inspectio The licensee did not identify any such documents/processes as proprietar. *

. '

AUGMENTED INSPECTION TEAM (AIT) CHARTER PALISADES POWER OPERATED RELIEF VALVE AND BLOCK VALVE FAILURE AND ASSOCIATED REACTOR TRIP AND SAFETY INJECTION EVENT ATTACHMENT 1 You and your team are to perform an inspection to accomplish the following: Develop and validate the sequence of events associated with the plant trip and safety injection that occurred on November 21, 1989 including the operation of the block valve and the power operated relief valve (PORV) during hydro testin.

Oetermi.ne the root cause for:

the pressurizer block valve failure to fully close after receiving a close signal from the plant operator including the valve control logic adequacy, limit and torque switch setting development and the adequacy of the calculated thrust and torque requirements for operation against expected differential pressure the opening of the PORV during the initial opening of the block valve and the subsequent (second opening) unexpected cycling of the POR the pressurizer block valve remaining open for a period of approximately three and one half minute.

Review the post modification testing of the PORV, the pressurizer block valve and related operational procedures for adequac.

Review the training provided to the operators prior to the implementation of the procedure for the pressurizer block and PORV for adequac.

Review the licensee proposed design change regarding the control circuitry on the block valve including the 50.59 evaluation for adequacy. Interview the plant operator directly involved in the event and determine the adequacy of operator action.

Review the initial design change package including the 50.59 evaluation for adequac.

Review the circumstances surro.unding the safety injection and the loading of the safeguards bus which caused the undervoltage condition and the diesel generator star ' *

TIME EVENT TIME (SECONDS)

PALISADES DEPRESSURIZATION - 11/21/89 Sequence of Events Description ATTACHMENT 2 1:13 The shift was authorized to perform control rod drive interlock testin To perform this test, the turbine and reactor protective systems were reset and the shutdown control rods were withdraw :06 PCS was at 2154 psi and normal operating temperature to perform a system hydrostatic tes :25:50 2:25:52 Note:

The log entries listed above establish the plant conditions that were germane to the event that follow Other activities were in progress, but did not pertain to the even PCS was at 2154 psi, Quench tank was at 0 psig, Quench tank level was at 75.4%, and Pressurizer level was 30.2%

Note:

Given subsequent indications, the block valve must have been given an 110pen 11 signal very near this tim PCS was at 2142 psi, Quench tank was at 1 psig, Quench tank level was at 78.9%, and Pressurizer level was 30.2%.

This was the first indication that the PORV and the block valve were ope The reactor operator had opened the block valve in order to pressurize the space between the block valve and the POR The operator had no advance information, either by precaution in the procedure or by previous training, that the PORV would open when the PORV was suddenly exposed to pressur Note:

The operators stated, in written statements contained within the post trip review and during interviews conducted subsequent to this event, that numerous attempts were made to close the block valv ' *

2:26:16 2:26:26 2:28:36

2:28:38 2:28:48 2:28:52

36 PCS was at 1758 psi, Quench tank was at 4 psig, Quench tank level was at 78.7%, and Pressurizer level was 28.5%.

Reactor trip due to activation of Thermal Margin Low Pressure (TMLP).

PCS was at 1669 psi, Quench tank was at 5 psig, Quench tank level was at 85.4%, and Pressurizer level was 28.4%.

PCS depressurization had stopped, which indicated the PORV was close The plant computers indicated the block valve ~as not close Note:

The block valve design contains a 11 seal in 11 circuit that requires completion of travel before direction can be changed. * As such, a cycle time of almost 40 seconds (20 seconds one way stroke time) was required to close the valv PCS was at 1738 psi, Quench tank was at 5 psig, Quench tank level was at 83.4%, and Pressurizer level was 29.4%.

The plant had recovered to these parameters PCS was at 1734 psi, Quench tank wai at 5 psig, Quench tank level was at 83.0%, and Pressurizer level was 29.3%.

This was the second opening of the POR One of the plant computers indicated that the PORV opened 6 times within 1 secon PCS was at 1605 psi, Quench tank was at 7 psig, Quench tank level was at 86.8%, and Pressurizer level was 29.1%.

Safety Injection signal was activated at this tim PCS was at 1565 psi, Quench tank was at 7 psig, Quench tank level was at 86.7%, Pressurizer level WP.S 28.1%.

The PORV appears to have seated a second tim The PCS pressure was starting to increase.

Attachment 2 Page 2 of 3

c *

2:29:18 208 PCS was at 1583 psi, Quench tank was at 6 psig, Quench tank level was at 85.9%, Pressurizer level was 29.4%.

The block valve indicated CLOSED for the first time since the event bega :56 Unusual Event declared because the PORV was challenge :05 SIS reset 3:34 License secured from the Unusual Event Note:

Normal operating practice was to heat up (and cooldown)

with the PORVs disarmed and the PORV block valves closed, extept to provide LTOP protectio This mode of operation was practiced since initial startu The updated FSAR describes this mode of operation.

Attachment 2 Page 3 of 3

Persons Contacted Consumers Power Company

1. G. B. Slade, Plant General Manager

1. R. M. Rice, Plant Operations Manager

1. J. G. Lewis, Technical Director

1. R. D. Orosz, Engineering and Maintenance Manager

1. W. L. Beckman, Radiological Services Manager J. L. Hanson, Operations Superintendent R. 8. Kasper, Mechanical Maintenance Superintendent K. E. Osborne, System Engineering Superintendent ATTACHMENT 3 R. M. Brzezinski, I&C Engineering and Maintenance Superintendent C. S. Kozup, Technical Engineer

1. D. J. Malone, Senior Licensing Analyst R. J. Frigo, Operations Staff Support Supervisor W. L. Roberts, Plant Projects Supervisor R. W. Smedley, Staff Licensing Engineer K. A. Toner, Plant Projects Superintendent J. P. Pomaranski, Project Manager, ESS Engineering R. T. Gilmore, Project Manager

1. T. S. Palmisano, Administrative and Planning Manager W. E. Garrity, Engineering Manager, ESS D. E. Engle, Staff Engineer P. M. Fitten, Staff Engineer

1. R. E. Mc Caleb, Site QA Manager

1. Indicates some of those in attendance at the December 15, 1989 Management Interview~