ML18095A486
| ML18095A486 | |
| Person / Time | |
|---|---|
| Site: | Salem  |
| Issue date: | 08/29/1990 |
| From: | Swetland P NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML18095A484 | List: |
| References | |
| 50-272-90-20, 50-311-90-20, NUDOCS 9009250150 | |
| Download: ML18095A486 (16) | |
See also: IR 05000272/1990020
Text
I*
Report No.
License
Licensee:
Facility:
Dates:
Inspector.s:
Approved:
U. S. NUCLEAR REGULATORY COMMISSION
REGION I
50-272/90-20
50-311/90-20*
Public Service Electric and Gas Company
P. 0. Box 236
Hancocks Bridge, New Jersey 08038
Salem Nuclear Generating Station - Units 1 and 2
June 28 - August 15, 1990
Thomas P. Johnson, Senior Resident Inspector
Stephen M.
P~ndale, Resident Inspector
Stephen T. Barr, Resident Inspector
Harold I. Gregg, Senior Reactor Engineer
Reactor Projects Section 2A tt?<t:ta_,_90
~
Inspection Summary: Inspection 50-272/90-20; 311/90-20 on June 28 - August 15, 1990
Areas Inspected:
Special inspection to review the circumstances surrounding,
and the licensee response to, the reactor trip of June 28, 1990 and the sub-
sequent identification of deficiencies in the main steam isolation circuitry
and potential deficiencies in the main steam isolation valves.
Results:
The inspectors identified one non-cited deviation concerning the
failure of the main steam isolation circuit to meet all the requirements of
IEEE Standard 279-1971 as committed to in the Salem Updated Final Safety Analysis
Report.
The inspectors also identified weaknesses in the licensee 1 s initial
knowledge of MSIV performance characteristics, in the preventive maintenance of
the valves, in the licensee 1 s tolerance of degrading valve conditions, and in
the lack of proper review by the licensee of the first attempt at fixing the
circuit deficiency.
The inspectors noted that Salem system engineering responded
well in defining the MSIV problem and in developing maintenance and surveillance
techniques to correct those problems.
Additional strengths were exhibited by
licensee engineering and management in the conservative postulation, analysis
and solutions of the various problems encountered with the MSIVs and their
isolation circuitry.
PSE&G was recognized as being open and thorough in their
communications with the NRC concerning the MSIV problems and their resolutions.
TABLE OF CONTENTS
PAGE
- 1.
Overview.............................................................
1
2.
Main Steam Isolation Valve Description and Operation.................
2
3.
Main Steam Isolation Valves (MSIVs) Circuit Design...................
3
4.
Regulatory Requirements..... . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4
5.
Sequence of Events..................................... . . . . . . . . . . . . . .
4
6.
Licensee Activities and Corrective Actions...........................
5
7.
NRC Inspection Activities and Results ................................ 10
8.
NRC Conclusions ........................................................ 13
- 9.
Exit Inte.rview ........................................ 1 ***************
14
..
1.
DETAILS
Overview
On June 28, 1990 at 12:32 a.m., Unit 2 automatically tripped from 75% power
following a loss of feedwater flow.
The event began with a failure of
non-vital 4kV/460V transformer No. 2F, which resulted in the loss of the
operating lubricating oil (LO) pump on each of the two operating steam
generator feed pumps (SGFPs).
The turbine governor controls for the No. 22
SGFP were also powered from the affected 460 volt bus.
The No. 21 SGFP backup LO pump automatically started following the loss of
the operating LO pump.
It did not, however, effectively restore LO system
pressure to prevent the automatic low LO pressure SGFP trip from occurring.
The No. 22 SGFP was also immediately lost, due to both the loss of the
operating LO pump and loss of turbine governor control.
The reactor auto-
matically tripped due to steam flow/feed flow mism~tch coincident with low
steam generator water level.
All three auxiliary feedwater (AFW) system
pumps, two motor driven and one turbine driven, automatically started fol~
lowing the trip as designed.
While responding to the reactor trip per emergency ope~ating procedures,
plant operators experienced one significant abnormality.
Approximately
eleven minutes following the trip, a control room operator attempted to
manually initiate a fast main steam line (MSL) isolati"on by depressing the
MSL isolation control panel pushbuttons in order to reduce the primary
system cooldown per emergency procedure direction.
Only two of the four
main steam isolation valves (MSIVs) indicated closed.
A slow MSIV closure
was then initiated, but no immediate results were noted.
About seven
minutes following the initial fast MSIV closure attempt, a second attempt
was made.
This time, the MSL isolation pushbuttons were maintained de-
pressed for several seconds and the control room console closed indications
lit for the remaining two MSIVs.
The unit was subsequently stabilized in
Mode 3 (Hot Standby).
Approximately 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> following the reactor trip, the licensee concluded
that two of the four MSIVs had not fully closed in response to the i~itial
manual MSL isolation attempt.
Followup investigation revealed an anomaly
within the MSIV circuitry by which, under specific conditions, both the
manual and automatic MSL isolation signals will not seal-in and thereby
not complete MSIV closure unless the signal (demand) remained until the
valves fully closed.
The licensee notified the NRC Operations Center of the reactor trip and
the subsequent findings in accordance with 10 CFR 50.72 reporting require-
ments .
2.
2
Main Steam Isolation Valve Description and Operation
Salem Units 1 and 2 each utilize four MSIVs manufactured by Hopkinsons
Limited of England (distributed in the U.S. by Atwood & Morrill).
The
valves are 27 inch parallel slide (double circular discs), reverse acting
gate valves, with two integral operating pistons and cylinders.
The bottom
of the gate holds the double circular discs and the gate has a 24-inch
diameter through-hole located above the discs.
When the gate is in the
full down position, the through-hole is aligned with inlet and outlet ports
and the valve is full open.
In this position, the bottom of the gate and
the double discs are in a lower body chamber below the valve through-port.
Closing the valve is accomplished by moving the gate to the full upward
position and the double circular discs are aligned with the i~let and outlet
ports, and through-flow is prevented.
The valve stem has a piston and cylinder arrangement within the valve bonnet;
system steam pressure above and below the piston balances the gate in its
full open and detent latched position, or in full closed and detent latched
position.
The detent mechanism of the valve is contained in a yoke assembly
attached to the valve stem. The yoke holds four spring and roller assemblies
that, wh~n the valve is in the full open or closed pos:it*ion, engage detents
in four lugs attached to the valve body (two lugs in the full closed posi-
tion and .two in the full open position). The force th~t the spring ex~rts
to keep the roller in the detent assists in keeping the* valve in the full
open or full closed position. The piston has a small equalizing orifice
and a small condensate drain pipe attachment.
During normal
operatio~,
the steam cylinder has equal steam pressure in each of the two chambers
because of the equalizing orifice.
Each MSIV has two air operated vent valves connected to the upper steam
chamber of the steam cylinder.
Vent valve position is controlled by a
solenoid valve, located in the air supply line to each valve.
The solenoid
valves allow air pressure to hold the vent valve in the closed position
unless an MSIV emergency fast closure signal is received.
Upon receipt of
an MSIV fast closure signal, steam evacuates from the upper chamber through
the two vent valves.
The resulting pressure inbalance provides the motive
power to move the piston and gate out of the detents and to the full upward
(valve closed) position.
The valves were required to fast close within
five seconds per Technical Specification requirements and design basis
assumptions.
An eight second closure time had been recently approved for
one operating cycle.
In order to improve MSIV closing times, the licensee
plans to implement a modification which would facilitate condensate drainage
from the top of the piston in the steam cylinder.
A hydraulic cylinder (utilizing hydraulic oil) and pump motor assembly
attached to the upper end of the stem is the only means to open the valve
and is an alternate means to close the valve.
It also acts as a hydraulic
snubber during fast closure.
The MSIV slow hydraulic stroke takes about
five minutes .
- .
3.
3
The MSIVs are designed to provide protection in the following cases:
For a MSL break inside containment, the MSIVs serve to isolate reverse
flow to limit the containment building pressure rise.
For a MSL break outside containment, the MSIVs serve as isolation to
prevent the uncontrolled blowdown of more than one steam generator.
To minimize the positive reactivity effects of the reactor coolant
system associated with an excessive primary system cooldown.
The MSIVs receive an automatic fast closure signal from the MSL isolation
actuation circuitry, which is generated by either high-high containment
pressure, or high MSL flow coincident with either low-low Tave or low MSL
pressure.
The only other U.S. nuclear plant with Hopkinsons MSIVs is D.C. Cook.
Main Steam Isolation Valves (MSIVs) Circuit Design
The MSIVs are manually operated by various means.
They are individually
slow-closed or slow-opened from the control room panels (bezels).
Fast-
closure is accomplished via individual MSIV fast-close bezels or through
individual MSL isolation bezels.
There are two safety-grade trains associ-
ated with the MSL isolation function, consisting of four pushbuttons per
train (two per MSIV).
The MSL isolation circuitry closes the MSIVs, the
MSI.V bypass valves, and the main steam line. drain valves.
Only the MSIVs
are normally open during power operation.
The automatic MSIV closure occurs
as a result of an MSL automatic isolation signal.
The MSL isolation circuitry was such that an open limit switch permissive
was required to allow a short duration isolation signal to fully close the
valves.
That is, if an MSIV was not fully open (with its open limit switches
not satisfied), the valves would close in response to both a manual and
automatic isolation signal only as long as the signal was present.
The
MSIVs are required to close within five seconds.
Included in the five
seconds is a 1.5 second time delay to allow for a shuttle v~lve in the
hydraulic system to reposition prior to MSIV movement.
Only the manual
and automatic MSL isolation functions were affected by this anomaly. The
manually initiated individual MSIV fast-closure actuation function (from a
separate bezel) was not affected.
The result of this anomaly was that
MSIVs that did not have open limit switches satisfied would not have fully
closed in response to a momentary (manual) pushbutton MSL isolation or for
an automatic MSL isolation signal of short duration (less than five seconds).
At the time of the event, the Salem operators had not been trained in this
peculiarity of the isolation circuit and, consequently, were not aware
that the isolation pushbutton would have to be held depressed until the
valve was closed for any valve that was off its full open seat .
..
4
The licensee had preliminarily concluded that this circuit design may be a*
generic NSSS vendor (Westinghouse) design.* The licensee had been in com-
munication with the vendor since June 29, 1990, in order to evaluate the
design basis of the circuit. It was determined that the logic design and
requirements were developed by Westinghouse, but the circuit design de-
velopment was left to the individual plant's architect and engineering
firm, which in this case was PSE&G themselves.
4.
Regulatory Requirements
The Salem Updated Final Safety Analysis Report (UFSAR) Section 7.3.1.1.6,
"Main Steam Isolation", states that the automatic actuation system is de-
signed to meet the requirements for protective systems as described in
UFSAR Section 7.2.1.
UFSAR Section 7.2.1.3, "Principles of Design", states that the reactor
trip system is designed in accordance with IEEE Standard 279-1971, "Cri-
teria for Protective Systems for Nuclear Power Generating Stations".
IEEE Standard 279-1971, Section 4.16, "Completion of Protective Actio_n
Once It Ls Initiated, states that the protective system shall be so de-
signed that, once initiated, a protective action at th~ system level ~hall
go to completion.
Return to operation shall require s~bsequent delib~rate
operator action.
UFSAR Section 10.3.2.2, "Main Steam Stop Valves", states that the MSIVs
are capable of closure as long as main steam pressure is in 100 psig.
Salem Unit 1 and Unit 2 Technical Specifications 3.7.1.5, "Main Steam Line
Isolation Valves", require that each main steam line isolation valve shall
be operable while in Modes 1, 2 and 3.
5.
Sequence of Events
Prior to the reactor trip on June 28, 1990, Unit 2 was holding power at
approximately 75% for reactor physics data collection, and was in a power
ascension status following completion of its recent fifth refueling outage.
The trip occurred at 12:32 a.m.
During the response to the trip, emergency
procedures directed the plant operators to initiate a MSL isolation to
prevent an excessive cooldown (Tave was less than 530 degrees F).
Prior
to manually initiating the MSL isolation, the senior reactor operator (SRO)
noted that the open limit indicator as displayed on the control room con-
sole was not illuminated for two MSIVs (21MS167 and 24MS167).
The reactor operator (RO) subsequently initiated the MSL isolation (fast
closure) at about 12:43 a.m. by momentarily depressing the MSL isolation
Train A and Train B pushbuttons (four per train).
The operators observed
that a closed indication was received on only two MSIVs (22MS167 and
23MS167).
The position of the remaining two MSIVs was not known.
Neither
the open nor the closed limit indicators were lit for those valves, which
6.
5
is the expected indication when the MSIVs are not full open and not full
closed.
The RO then attempted a slow closure for 21MS167 and 24MS167 which
appeared to be functioning properly, however, the slow closure from fully
open to fully closed takes approximately five minutes.
At about 12:50
a.m., the SRO directed the RO to attempt a fast closure of the two MSIVs
from the same MSL isolation pushbuttons by maintaining the pushbuttons
depressed for several seconds.
This time, closure indication was received
on two separate console valve position indications.
At 12:50 a.m., seven
minutes following the first closure attempt, 21MS167 and 24MS167 valves
were closed.
Following the trip, Tave reached a value of 520 degrees F.
The primary
system cooldown was attributed to high AFW system flow rates, *the low decay
heat load associated with a reload core, and the roughly ten minutes it
typically takes to reach the procedural step which requires operators to
verify the value of primary system average temperature.
The unit was sub-
sequently stabilized in Mode 3 (Hot Standby), and the appropriate 10 CFR
50.72 notifications were made.
Licensee Activities and Corrective Actions
Following the reactor trip, the licensee performed a post-trip review, in
accordance with Administrative Directive No. 16,
11 Post Reactor Trip/Safety
Injection Review.
11
Also, in order to fulfill the PSE&G Nuclear Department
commitment to perform an independent review of each reactor trip, the Salem
General Manager convened a Significant Event Response Team (SERT), per
Nuclear Administrative Procedure NC.NA-AP.ZZ-0061(Q),
11Significant Event
Response Team Management.
11
The corrective actions recommended by the
post-trip review included resolution of the cause of the loss of the 2F
4kV/460V transformer, and of the problem with
11 closed
11 indication for the
21MS167 and 24MS167 valves.
The initial direction taken by the SERT also
centered around the 2F 4kV/460V transformer, along with the normal in-
dependent trip follow up.
On June 29, 1990, the licensee ~onfirmed that the 21MS167 and 24MS167 had
not fully closed following the reactor trip.
The licensee concluded that
only the manual circuit was affected.
A Station Operations Review Committee
(SORC) meeting was conducted on June 29, 1990.
During the meeting, it was
questioned whether the automatic MSL isolation logic was affected, however,
the presenter stated that the automatic circuit was not similarly affected.
The General Manager subsequently authorized the unit to escalate from Mode
3 to Mode 2 in order to perform the MSIV closure surveillance test to
verify the valve position indications and to conduct circuitry testing to
verify the manual MSL isolation design anomaly theory.
Prior to escalating
modes, the licensee participated in a telephone conference call with NRC
Headquarters and NRC Region I personnel to discuss the trip response and
the intended testing plan.
During the call, it was determined that a
11 spike 11 signal received in the automatic MSL isolation circuitry, with any
MSIVs off their full open limit, also would result in those affected valves
not fully closing. This caused the NRC to question the Salem commitment to
- '
..
6
IEEE 279-1971, which requires that a protective system be so designed that,
once initiated, a protection actfon at the* system level shall go to com-
pletion.
With this concern raised about the manual and automatic MSL
isolation actuation logic, the licensee suspended their plans for Mode 2
testing and convened a team of engineers from the Engineering and Plant
Betterment (E&PB) Department to determine whether the MSL isolation circuit
was in compliance with IEEE 279-1971.
The engineering team concluded that
the MSL isolation circuit did not comply with the IEEE 279 requirement any
time an isolation was actuated with a MSIV off of its full open seat.
This condition exists any time a MSIV has drifted off its open seat during
normal plant operation, as had been the case on June 28, 1990 or any time
a MSIV is being opened from the closed position, as is the case during a
plant startup.
Once the concerns were raised in regard to MSIV full closure, the licensee
determined that Unit 1 was similarly affected and implemented compensatory
measures at that unit, which was operating at power.
Among these measures
were briefings of all oncoming operating crews to inform them of the pos-
sibi~ity of an MSIV not going fully closed under specific conditions and
developing an operating log to verify a.nd document that all MSIVs indica*ted
fully open.
Salem Unit 2 was taken to Mode 4 (Hot Shutdown) on June 30,
1990 in order to accomplish a system engineering inspe~tion and maintenance
plan for .the MSIVs, including the valve instrumentation and actuating: de-
vices.
The plant remained in Mode 4 while the E&PB engineering team con-
tacted Westinghouse and developed the necessary modification to the M'SL
isolation manual and automatic actuation logic.
The E&PB engineering team eventually determined that the cause of the
failure of the MSIVs to close was a defect in the MSL isolation logic.
The seal-in function that ensures that the valve goes fully closed was
µrecluded from functioning due to the reset loop in the circuit overriding
the operate loop in the manual and automatic actuation modes if the MSIV
was not initially fully open.
The licensee corrected the problem by re-
versing the position of the operate and reset coils in the isolation cir-
cuitry.
The modification ensured that the actuation signal is sealed in
and the MSIV will go full closed, even if the valve was not fully opened
at the initiation of the isolation signal.
The modification was approved
by a SORC meeting on July 3, 1990 and subsequently installed on July 4,
1990.
Continuity and logic functional tests were completed on the new
circuit on July 5, 1990, while the plant was in Mode 4.
These tests showed
the modification to be successful in closing the valve regardless of in-
itial MSIV position.
However, reversing the reset and operating coils was
found to prevent the automatic reset of the isolation circuit.
The circuit
not resetting prevented any subsequent reopening of the valve due to the
close signal remaining sealed in, which precluded any hydraulic operation
of the valves.
E&PB engineering proposed that this problem could be re-
solved by manually resetting the circuit subsequent to any manual or auto-
matic MSL isolation signal. This proposal was satisfactorily tested,
approved by SORC and implemented into all necessary procedures.
E&PB was
tasked to study the relevant circuitry and determine if a more suitable
7
long-term solution was available.
In addition, E&PB engineering stated
that an in-depth analysis of this and other similar logic circuits would
be undertaken in order to determine if any other undetected flaws might
exist. Subsequent to the circuit modification and procedure revision, the
licensee participated in a telephone conference call on July 6,1990, to
inform the NRC of their corrective actions and their intention to proceed
with a Unit 2 startup over that weekend.
During the forced outage, the licensee had measured and visually inspected
all Unit 2 detent mechanism positions.
These activities found the valves
out of the detent open position in varying amounts for three of the four
Unit 2 MSIVs (21MS167 out by 3/8 inch, 22MS167 out by 1/4 inch, and 24MS167
out by 3/4 inch).
Also, one detent had a damaged roller (21MS167) and one
detent was jammed open (24MS167).
Subsequent to finding the damaged roller
and detent mechanisms, the licensee mechanically locked open the four MSIVs
and removed all eight roller assemblies in order to examine their internals.
Two damaged assemblies were replaced, and the other six were regreased and
checked for the proper spring constant (
11 K
11 ).
After the roller assemblies
were reinstalled, the licensee checked and adjusted the limit switch ~osi
tions on each MSIV. The limit switches were adjusted to ensure the valve
hydraulic motor was run long enough to allow definite latching by the
rollers and detents, and to ensure positive indication of valve full open
and full closed position was received by the isolation circuitry and dis-
played in the control room .
A similar Unit 1 MSIV inspection was also conducted which found one valve
(14MS167) slightly off (1/4 inch) its open detent position.
The detents
were not removed and refurbished because the unit was at power.
The in-
spector was advised that the proper detent positioning was made on all
MSIVs and that specific maintenance instructions were being developed to
define the detent position requirements.
In order to maintain Unit 1 at power, the licensee conducted an evaluation
to justify the continued operation of Unit 1 without implementing the con-
trol circuitry modification.
The evaluation concluded that Unit 1 could
be safely operated due to the recent history of Unit 1 MSIVs not drifting off
their open seat and due to the adequacy of the compensatory measures taken
at the unit.
The evaluation also concluded that the logic circuitry should
be modified at the next Unit 1 outage in order to enhance the fault tolerance
of the circuitry should a valve drift off its open seat.
The justification
for continued operation was accepted by Salem management at a SORC meeting
on July 3, 1990.
During plant startup on the morning of July 9, 1990 in Mode 2, the MSIVs
were fast closure tested while in Mode 2 (Startup), and all the MSIVs closed
within Technical Specification time limits, and the new procedures were
correctly performed by the operating crews.
An additional problem was
identified, however, during the startup and MSIV fast closure testing.
During the fast closure testing of the MSIVs, the licensee stationed an
operator in the vicinity of the MS169 and MS171 MSIV vent valves to verify
.,
8
that steam exhausted from both vent valves when the MSIVs are fast closed.
The operator noted that steam continued to exhaust from the vents even
after the MSIV had gone closed.
Previously, the vent valves reclosed upon
MSIV closure, preventing any further steam release.
This abnormal condi-
tion was investigated by station engineering, and it was determined that
the vent valves staying open was another result of the logic circuit not
automatically resetting after the MSIV closed.
It was further determined
that the vent valves did close when the maintenance technician manually
reset the circuit per the new procedures.
This was the second time that
the recent circuit modification had produced a result that could have been
predicted but was not. Consequently, the licensee decided to return the
plant to Mode 4 while an engineering team was convened to fully analyze
the MSIV isolation circuit, the modification that had been made, all possible
consequences of the modification, and any new changes that might be required.
The engineering team conferred with the vendor, Westinghouse, and concluded
that, with certain procedure revisions, continued operation of Salem Unit
2 could be safely justified.
Plant management, however, was concerned with the additional actions now
requ~red of the plant operators to restore the MSIVs to normal following a
MSL isolation.
Of particular concern was the steam generator tube rupture
accident scenario.
With Unit 2 modified as it was, a radiological release
path would exist from the affected steam generator through its vent valves
until the circuit was manually reset and the vent valves could close.
The
licensee weighed this concern against the initial concern of an isolation
signal occurring with one or more of the MSIVs having drifted off its open
seat and the signal not lasting long enough to drive that valve fully shut.
Plant management concluded, on the basis that Unit 2 MSIVs had been mech-
anically reworked such that drifting should not be as frequent and that
any accident requiring MSIV closure would generate a signal long enough to
fully close any drifted valve, that the original circuit design of Unit 1
was more conservative than the modified circuit at Unit 2.
As a result of
this conclusion, the licensee decided to return the Unit 2 MSL isolation
circuitry to its original configuration and develop a justification for
continued operation (JCO) that would cover both Salem units.
Realizing
that the original circuit design still did not meet IEEE 279 requirements,
' E&PB engineering remained tasked to redesign the circuit so that it will
meet all IEEE 279 requirements while still maintaining the automatic fea-
tures of the present circuit.
On July 13,1990, the licensee presented the
proposed circuit resolution to SORC and subsequently to the NRC in a tele-
phone conference call involving Region I and NRR.
With SORC approval and
NRC concurrence, the licensee proceeded with a new design change package
(DCP) to remove the modifications made to the Unit 2 circuit and the pre-
paration of the required JCO.
On July 18,1990, the DCP that would remove the modification and restore
the previously installed Unit 2 isolation circuit to its original configu-
ration was presented to the SORC, who recommended approval.
During the
preparation of the 10 CFR 50.59 Safety Evaluation and the JCO, however, an
E&PB mechanical and electrical engineering review identified a new problem,
9
in which both the Unit 1 and Unit 2 MSIVs may not meet design basis closure
requirements.
The design basis for the MSIVs, as stated in FSAR Section
10.3.2.2, is that the valves will close within the Technical Specification
required fast closure times as long as main steam pressure is in excess of
100 psig.
While studying MSIV performance in order to justify plant opera-
tions with the original isolation circuit, the licensee preliminarily con-
cluded that a higher steam pressure (approximately 250 psig) is actually
required to ensure valve fast closure.
Postulated steam line break accidents
are assumed to reach pressures less than 250 psig before the MSIVs would
be fully closed.
Due to this determination, Unit 1 was shutdown on July 22, 1990 and reached
Mode 4 on July 23, 1990.
Both plants were maintained in Mode 4 while the
licensee further evaluated the preliminary conclusions and investigated
corrective actions.
On July 23, 1990, PSE&G participated in a telephone
conference call with NRC Headquarters and Region I personnel to inform
them of the current plant status and of the steps that were being taken to
resolve the MSIV problems.
For the next two and one half weeks, PSE&G mechanical engineering analy~ed
all steam line break models to determine the resultant steam line pressures.
At the same time, PSE&G worked with Hopkinsons and Atwood & Morrill to
determine more accurately the steam pressure required to close the *MSIVs.
On July 30, 1990 a ~onference call was conducted to inform the NRC of the
progress being made.
Within a week of that call, PSE&G determined that
the steam pressure required to initiate MSIV closure was approximately 100
psig and that all MSIVs would close for all main steam line breaks analyzed.
While the mechanical engineering group was analyzing actual valve perform-
ance, the Instrumentation & Controls engineering group developed a new
modification for the MSL isolation circuit.
The new modification involved
replacing the open limit switch* relay in the circuit reset loop with a
closed limit switch relay.
This change prevents a valve that has drifted
off of its open seat from initiating a circuit reset and preventing valve
closure.
The new change also provides a seal-in function in the circuit
because the reset loop now cannot be energized until the MSIV has gone
fully closed.
This modification was presented to a Salem SORC on July 31,
1990 and subsequently installed in both units.
With all modifications and analyses completed and approved by SORC, PSE&G
made a conference call to NRC Headquarters and Region I on August 10, 1990
to present their findings and conclusions on the MSIV issue and their plans
for plant restart.
Subsequently, the licensee made preparations for a
parallel startup of Units 1 and 2.
Plant heatup had already been started,
and both units were in Mode 3 by August 13, 1990.
Both units developed
problems apart from the MSIVs during power ascension.
Each unit
eventually reached Mode 2, and each unit's MSIVs performed satisfactorily
during the fast closure testing in that mode.
At the conclusion of the
10
inspection period, Salem Unit 2 was at power and operating satisfactorily,
and Unit 1 had returned to and was remaining in Mode 5 awaiting reactor
coolant pump motor replacement.
7.
NRC Inspection Activities and Results
The inspectors reviewed the applicable regulatory requirements and licensee
commitments, operator logs, post-trip review data and plant drawings.
Discussions were held with licensee technical, maintenance and operating
personnel involved with the events.
On June 29, 1990, the inspectors par-
ticipated in the telephone conference call held between NRC and licensee
personnel to discuss issues related to the events.
In addition, the in-
spector performed several walkdowns to observe the MSIVs and vent valves
located in the Unit 1 and Unit 2 penetration areas, and the vent valve
discharge piping located on the roof.
Following the reactor trip, the licensee pursued the MSIV problem, although
it had been characterized by personnel on the day shift as an indication
prob~em.
Information relative to the two MSIVs drifting off the open limit
switches was apparently not communicated to the appropriate day shift per-
sonnel.
As discussed previously, a control room operator observed that
the MSIVs that did not close after the trip had drifted off the full open
position immediately following the trip and prior to the MSL isolation.
A
separate incident report was written which reflected the observed.drift
conditions.
This report was apparently not reviewed by personnel reviewing
the trip response.
The lack of this knowledge resulted in the MSIV problem
being perceived to be an indication-only problem and therefore, not aggres-
sively pursued.
This lack of knowledge of the event details also apparently
delayed the involvement of the system engineers.
About 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> following
the trip, system engineering concluded that the valves had not fully closed,
utilizing information relative to valve position prior to the MS~ initiation
and by review of circuit drawings.
During the June 29 conference call, the inspector questioned the numerous
Unit 2 control room operator log entries relative to MSIV drifting and
manual reopening, however, licensee personnel involved in the conference
were unaware of the entries. The inspector identified that on June 24,
1990, 23MS167 and 24MS167 drifted off the open limit and was manually re-
opened by plant operators from the control room.
On June 25, 1990, 21MS167
drifted and was reopened.
MSIVs have historically drifted off their open seat, especially at Unit 2.
Position drifting had previously been attributed to vent valve leakage
problems.
Prior to the March 30, 1990 shutdown for the refueling outage,
two Unit 2 MS!Vs were drifting off their full open position, as observed
in the control room, roughly three times a day.
Excessive leakage was
then observed at the vent valves and roof vents.
After the June 25, 1990
startup from the outage (after all vent valves were repaired), all MSIVs
drifted at least once during the short time prior to the June 28, 1990
trip.
There have also been recent condensate buildup problems which have
11
resulied in a Technical Specification change to increase the allowed valve
closing stroke times.
The inspector found* that the recent valve drifts
were not communicated to the system engineer by operations, nor did the
system engineer become aware of the problems by independent log review.
This is an example of licensee personnel accepting potentially significant
off-normal conditions without further question because the problems had
. existed for a long period of time and had become an accepted discrepancy.
On June 29, 1990, the inspector observed the MSIVs and vent valves in the
outer penetration of Unit 1.
Unit 2 valves 22MS167 and 24MS167 were in
the closed position and the stem indicator pointers were on the shut scale
mark.
The detent posts were observed to be protruding through the stop
bar, but the detent could not be verified to be in the latched position.
Through discussions with the system engineers, latch position verification
is performed by observing that the detent post protrudes through the stop
bar, but no measurements or alignment to any mark was made or was required
by procedure.
The four vent valves associated with the MSIVs in the same area were ob-
served to be leaking, as ~videnced from the water discharge from the drilled
leak orifices in the vent pipe downstream of the valve5.
These valve~ had
just undergone an overhaul, and the inspector noted that leak testing; is
not a requirement for these va 1 ves fo 11 owing such an activity.* The i'n-
spector also expressed concerns that this type of valve may not be the
most leak tight, and its actuator force in the closed position may not be
large enough to limit the leakage to an acceptable amount (less than the
piston orifice flow).
Observations of the roof vents were also made and
there was no evidence of vapor from these valves.
Similar walkdown observations were made of the Unit 1 outer penetration
MSIVs and vent valves.
Again, the vent valve5 were observed to be leaking
and two of the roof vent pipes had substantial vapor flows.
The inspector questioned whether specific detailed procedures were his-
torically used to adjust the MSIV limit switches and/or detents.
The lic-
ensee stated that they had not used any procedure while setting the limit
switches/detents, and that those activities were assumed to be within the
knowledge of the technicians.
The inspector, however, expressed concern
that the MSIVs are unique valves, using several limit switches (two open,
two closed) which must be set and coordinated with detent position.
The
inspector found that detent position was not specifically considered during
limit switch adjustments.
The inspector also found that the detent mechanisms have not been maintained.
During an inspection of the Unit 2 detents, the licensee found that the
detent mechanisms' internal oil had burned off and allowed the internal
detent components to degrade.
The licensee subsequently determined that
the oil temperature rating was insufficient relative to the operating tem-
peratures in the MSIV penetration rooms.
Also, there were only two spare
.
'
12
detent mechanisms in stock (there are four detent mechanisms per MSIV).
The licensee also found detent mechanism iTiternal 0-rings to be brittle.
New 0-rings were not in stock, however, the licensee was able to procure
new replacement a-rings.
The inspector determined through discussions with the system engineers
that the MSIVs had never been disassembled.
Further, the detent mechanisms
had never been disassembled or maintained via a preventive maintenance
program.
The inspector found that the bottom of the lower body cavity was
removed on several valves in the early 1980 1 s to view the disc seats, but
no work was performed.
Also, one hydraulic actuator was removed from a
valve in the same time frame.
During the past outage, all Unit 2 vent
valves were overhauled.
The solenoid valves, check valves, and relief
valves on the hydraulic units were also rebuilt at the same time.
The
inspector noted that the vent valves have no leakage test requirements
following repair.
Also, there had been no surveillance or maintenance
process to verify the precise detent latch position location.
Based on observations of the valves and review of the assembly drawings,
the Hopkinsons MSIVs appear to be sturdy and well constructed valves.
Since the valves have never been disassembled, there i~ little knowledge
of the internal wear of the seats, the condition of the piston orifice and
drain tube, and other important internal parts.
The v~lves have unde~gone
numerous actuations due to trips and several surveillance fast-closure
strokes with generally favorable results, except for those related to; the
noted vent valve leakage and condensate problems that required the valve
closing times to be increased.
It was only subsequent to this event that the licensee in fact took many
of the above mentioned measures, such as a more precise setting of the
MSIV limit switches, int2rnal examination of the roller mechanisms, the
use of a temperature resistant grease in the rollers, and verification of
proper roller-detent alignment. The licensee also performed these measures
on the Unit 1 MSIVs during the Unit 1 outage, while the engineering analyses
were being performed. System engineering has been tasked with implementing
the preventive maintenance measures into station surveillance procedures
and making them recurring tasks in order to optimize future valve perform-
ance.
The inspectors monitored the development of the procedures and wit-
nessed the initial implementation of many of them during the restoration
of Units 1 and 2.
Coincident with the maintenance and evaluation of the MSIVs, system and
E&PB engineering pursued the resolution of the MSL isolation circuit de-
ficiencies.
The resident inspectors participated in the initial telephone
conference calls which raised the concerns about the MSL isolation circuit
and followed the licensee's first attempt at modifying the circuit.
An
8.
13
inspector was present during the July 9, 1990 startup attempt and witnessed
the plant operators' and system engineer 1 s recognition of the abnormal
event when the MS169 and MS171 vent valves remained open.
From this point
on the resolution of the MSIV problem centered on PSE&G and vendor engi-
neering analysis of valve performance, circuit design and potential accident
scenarios.
The resident inspectors closely monitored the licensee 1s pro-
gress in arriving at the eventual solution to the MSIV and associated cir-
cuit problems, and at least one inspector was present at every NRC con-
ference call and Salem SORC meeting previously noted in Section 6.0 above.
An inspector was also present to witness the final successful fast closure
testing of MSIVs at the end of the inspection period.
The inspectors noted
that the licensee was open and forthright in keeping the NRC
inspectors informed of both the progress and setbacks experienced
during the resolution of the problem.
NRC Conclusions
The majority of the problems and concerns with the MSIVs and associated
isol~tion circuit discussed in this report were discovered subsequent to
the Salem Unit 2 reactor trip of June 28, 1990.
Through the testing,
operating experience and engineering analysis described in section 6.0 of
the report, the licensee achieved a satisfactory resolution of the problems.
Although initial identification of the problem and the first attempt*at
fixing it were faulty, the licensee's thoroughness and conservative approach
thereafter produced an acceptable solution.
Weaknesses were noted in the licensee 1 s prior knowledge of MSIV charac-
teristics, their effect on valve performance, and in the inadequacy of
relevant maintenance and surveillance procedures to assure proper valve
operation.
This lack of a thorough understanding of the MSIVs and of
~roper valve maintenance contributed to another identified weakness, the
licensee
1 s tolerance of degraded MSIV performance, as evidenced by the
recurrent valve drifting events.
Following the discovery of the circuit
deficiency, the engineering team again displayed an insufficient
understanding of how the MSIVs and the circuit function by recommending
the reversal of the operating and reset loop coils.
This modification was
made too hastily, without the proper engineering review and verification,
and resulted in the introduction of more complex and potentially more
severe concerns.
PSE&G was also slow in understanding the requirements of IEEE 279 as they
pertained to the MSL isolation circuit.
The fact that the MSIVs were not
assured of going completely shut if a valve was not initially fully open
when an isolation signal was initiated was a failure to comply with IEEE
Standard 279-1971, Section 4.16, which was committed to in the Salem UFSAR.
However; due to the licensee's self-identification of the non-compliance
and their timely and considerate correction of the deficiency, this failure
to meet a commitment is classified as a non-cited deviation.
(NON 272/90-20-01).
9.
14
~nee the MSIV problems were identified and better appreciated, Salem system
engineering responded well by conducting the proper testing and evaluation
of actual MSIV performance and by developing the new maintenance and sur-
veillance procedures needed to deter future drifting and position indication
problems.
Also noteworthy was the plant operators* post-trip performance
in recognizing and overcoming the failure of the 2 MSIVs to close and the
assistance they provided to engineering in defining what actually had
occurred on June 28, 1990.
After the initial circuit modification was
found to be unsatisfactory, the licensee's engineering effort to resolve
the matter was more deliberate and thorough, as indicated by the in-depth
analysis of all pertinent accident scenarios and by engineering's question-
ing attitude which initiated the full review of actual MSIV performance.
While the engineering team was pursuing these tasks, PSE&G management dis-
played a conservative and safety conscious attitude by shutting down Unit
1 and maintaining both units shut down until a complete and proper resolu-
tion to all problems had been achieved.
The inspectors also noted that
the licensee, both management and staff, was open and cooperative with the
NRC resident staff and Headquarter personnel in the communication and re-
solution of the issue.
Exit Interview
An exit meeting with members of Salem plant management and staff was con-
ducted on August 22, 1990.
The inspectors discussed the inspection and
presented its findings and conclusions to the licensee.
The licensee had
no major questions and expressed their appreciation for the NRC 1 s coopera-
tion throughout the inspection period.
Based on Region I review and discussions with PSE&G, it was determined
that this report does not contain information subject to 10 CFR 2 restric-
tions.