ML18101A611
| ML18101A611 | |
| Person / Time | |
|---|---|
| Site: | Salem, Hope Creek  |
| Issue date: | 03/24/1995 |
| From: | Calvert J, Rogge J, James Trapp NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML18101A610 | List: |
| References | |
| 50-272-95-03, 50-272-95-3, 50-311-95-03, 50-311-95-3, 50-354-95-02, 50-354-95-2, IEIN-95-010, IEIN-95-10, NUDOCS 9504030215 | |
| Download: ML18101A611 (19) | |
See also: IR 05000272/1995003
Text
- .'
DOCKET/REPORT NOS:
LICENSEE:
FACILITIES:
DATES:
INSPECTORS:
APPROVED BY:
9504030215 950324
ADOCK 05000272
a
U. S. NUCLEAR REGULATORY COMMISSION
REGION I
50-272/95-03
50-311/95-03
50-354/95-02
Public Service Electric and Gas Company
Salem Nuclear Generating Station,
Hope Creek Nuclear Generating Station, Units 1 and 2
Hancocks Bridge, N.J.
February 4-17, 1995
q~J1lti~4ff
J
Ca vert, Reactor Engineer
ectrical Section
ivision of Reactor Safety
v/J ;!, /k-1&,f ~
/'/: im rapp,
earn ea er
Division of Reactor Safety
3/2,,f/2(
' Date
3/z<f/Jr
Date
- ' '
EXECUTIVE SUMMARY
The purpose of this inspection was divided into two separate activities.
One
activity was performed at the Salem Units 1 and 2, and the other performed at
the Hope Creek plant.
The first week of the inspection reviewed Salem's actions taken in response to
the potential loss of an automatic engineered safety features equipment
actuation signal and, the subsequent problems that occurred with the solid
state protection system (SSPS) power supplies when attempting to correct this
deficiency. The potential loss could occur as a result of a postulated main
steam line break or seismic event in the turbine building which could cause
electrical faults in some SSPS non-class lE input signals. The faults could
disable the power supplies in one or both of the SSPS trains. This could lead
to the loss of the ESF actuation signals.
(NRC Information Notice 95-10)
The second week of the inspection primarily focused on the Hope Creek
technical department, their program initiatives, implementation of the
temporary modification program, and performance in the resolution of technical
issues and problems.
The inspectors found that the Hope Creek technical department was adequately
staffed. Communications between the technical department and operations staff
were strong and effective. The experience level of the system engineers was
strong and the system engineers were knowledgeable of their assigned systems
and technical areas of expertise. The system engineering walkdown program was
a good initiative and was being effectively used to identify equipment
deficiencies. Technical department management was proactive in developing
initiatives, such as the system readiness and work around list to enhance
station performance.
The Hope Creek temporary modification program implementation was good.
Installed temporary modifications were installed in accordance with station
administrative procedures and safety evaluations were thorough.
The total
number of temporary modifications installed in safety-related systems was low.
The quality of Hope Creek root cause evaluations reviewed were inconsistent.
Some of the root cause evaluations reviewed were thorough, while others were
poorly documented and lacked technical rigor.
For example, a root cause
evaluation completed in 1993, regarding the failure of safety auxiliary
cooling system (SACS) room cooler valves, was narrowly focused and was
unsuccessful in preventing repeat valve failures.
The root cause evaluation
documentation provided for the most recent SACS valve failures was not
complete and root cause analysis (RCA) techniques required by procedures were
not rigorously used.
The management acceptance rate of root cause analyses
(RCA) done by the technical staff was low and management standards for RCA
were not clearly defined. Technical department management was aware of the
weaknesses in the RCAs, required for the corrective action program, and was in
the process of making significant changes to improve the program .
ii
The suitability of the Hope Creek reactor protection system {RPS) loads to
possible change in total harmonic distortion {THO) caused by the change in the
alternate supply transformer was not determined and is an unresolved item.
The licensee plans to measure the THO on the bus, while utilizing the
alternate supply, and will determine the suitability for the RPS loads.
{Unresolved Item 50-354/95-02-01)
The root causes for the Hope Creek plant trip caused by the faulty logic in
the overfill protection system of the digital feedwater control system were
well documented, reasonable, detailed, and valid.
The assessment of the performance associated with the Salem potential loss of
automatic ESF equipment actuation signal is given below.
1.
The E&PB engineering staff showed timely and appropriate defense of the
design basis by their research of the actual Salem design status when
information was given to them verbally by another utility. Management
acted promptly on the information presented to them by the engineers.*
This was an example of an appropriate safety-conscious decision process.
2.
Management involvement and direction was evident by the quick dispatch
of engineering specialists on a full 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> day coverage for
troubleshooting, analysis, and corrective action when the SSPS power
supply problems occurred.
3.
The troubleshooting work process was planned, controlled, and
documented.
The analysis of failures was accurate, detailed,
documented, and technically sound.
The Salem system engineers provided
accurate analysis in the operability determinations of the SSPS.
iii
DETAILS
1.0
PURPOSE AND SCOPE (IP 37550)
The purpose of this inspection was divided into two separate inspection
activities. The first week of the inspection reviewed Salem's actions taken
in response to the potential loss of an automatic engineered safety features
(ESF) actuation (NRC Information Notice 95-10) and the subsequent problems
that occurred with the solid state protection system (SSPS) when attempting to
correct this deficiency (NRC Information Notice 95-10, Supplement 1). The
second week of the inspection primarily focused on the Hope Creek technical
department, including their program initiatives, implementation of the
temporary modification program, and performance in resolution of technical
issues and problems.
The inspectors also reviewed two Hope Creek plant
modifications.
2.0
HOPE CREEK TECHNICAL DEPARTMENT
2.1
Organization and Responsibilities
The Hope Creek Technical Department organization is comprised of a technical
manager, 6 technical engineers (supervisory level engineers), 32 system
engineers (SEs), 4 reactor engineers, and an administrative and technical
support staff. The function of the technical department is to provide
technical support to station personnel through the system engineers who serve
as system experts. These activities include system performance evaluation,
procedure development and review, initiation of corrective actions, project
team members for modifications, development of temporary modifications, and
the performance of root cause analyses and safety evaluations. The SEs also
provide assistance to operations department on equipment and system
operability determinations. At the time of the inspection, the administrative
and the engineering positions were fully staffed with the exception of 3
system engineers positions in the balance of plant area.
The inspectors
concluded that the technical department is adequately staffed at this time.
The system engineers that were interviewed were cognizant of their assigned
responsibilities. All of the system engineers interviewed had extensive
nuclear experience, including many with previous experience with reactor
vendors or architectural engineering firms. Several system engineers were
observed in attendance at the plant morning status meetings where they
provided information regarding the system for which they were responsible.
Licensed operations personnel were cognizant of system engineer assigned
responsibilities. The operations personnel interviewed stated that the system
engineers provided strong support of plant operator issues and concerns.
System engineers were observed in the control room and in the plant when
issues such as LCO maintenance on their assigned systems was being performed.
Specifically, strong communication was observed between the reactor core
isolation cooling system engineer and operations personnel during RCIC
maintenance activities. The inspectors concluded that the system engineers
communications with other station personnel were strong and effective .
2
2.2
Techn;cal Department Programs
Three programs implemented by the system engineers were reviewed to assess the
quality. The programs reviewed were the system engineers walkdown program,
operations work-around elimination program, and the system readiness rev;ews.
The inspectors accompanied a system engineer on a routine monthly walkdown of
the system engineer's assigned systems.
The systems walked down were balance
of plant system located in the turbine building. The system engineer was very
knowledgeable of the system performance. Crit;cal system performance
parameters were recorded and the system engineer was cognizant of past
performance trends.
The system engineer frequently commun;cated with
operat;ons personnel during the walkdown to gather additional ;nformation
regarding system performance.
The walkdown program expectations are
documented in a technical department d;rective. Documented findings of other
system engineer walkdowns were reviewed and were found to provide good
feedback on system performance issues. The inspectors concluded that the
system engineering walkdown program was a good initiative and was being
effectively used to identify equipment deficiencies.
The Hope Creek Technical Department has recently developed a operations work-
around elimination program.
The program administration guidelines are
described in Station Directive SA-SD.ZZ-27.
The program is designed to
enhance the timeliness of resolution of issues that could complicate operator
response to plant transients that distract control room personnel from normal
duties. The plant operators are responsible to identify those deficiencies to
be included in the work-around program.
The issues are tracked using the
performance improvement request system (PIRS).
The technical department
engineers evaluate the work-arounds and initiate corrective actions. This
program was recently initiated and the effectiveness of the implementation of
the program was not reviewed.
However, the inspectors concluded that this
program is a positive effort by the technical department management to provide
enhanced support for operator concerns.
The Hope Creek technical department has recently developed a system readiness
review of important systems. A system readiness report includes several
important parameters including corrective maintenance work orders, incident
reports, temporary modifications, and engineering concerns.
The reports are
computer generated and provide a sound bases for establishing the readiness of
the system.
Five systems readiness reports have been generated.
The system
engineers and technical department management demonstrated a commitment to
completing the system readiness reports for other key systems.
The system walkdown program was being implemented in a quality manner.
The
work-around and system readiness programs were good initiates. The inspectors
concluded that the implementation of these initiatives demonstrated the
Technical Department management's strong commitment to enhancing department
performance .
3
3.0
The temporary modification {TM) process was reviewed to verify that the
installed temporary modifications do not degrade the function of plant safety
systems.
The inspectors reviewed TM administrative controls, the detailed
design information, and conducted field observation of several installed TMs.
The engineering management attention to address long-standing design
deficiencies, temporarily resolved by installing TMs, was also reviewed.
The TMs are administratively controlled in accordance with administrative
procedure NC.NA-AP.ZZ-0013(Q) "Control of Temporary Modifications." The
procedures provided detailed instructions and designated the responsible
individuals for TM installation. Station Operations Review Committee (SORC)
review is required of all safety-related TMs prior to installation. The
administrative guidance provided for TM installation was detailed with a
particular strength noted in the design criteria specifications.
The inspectors reviewed the following temporary modifications:
1.
TM 94-039 "Disabling the EOC/RPT Al arm"
2.
TM 94-026 "MSIV Seal System Draining"
3.
TM 94-027 "Adjustment of Reset Point on IEG-FSL-2544D."
The inspectors verified that the temporary modifications reviewed were
installed in accordance with the installation instructions. The design of the
temporary modifications reviewed were technically sound and did not degrade
the function of plant safety systems.
The safety evaluations were thorough
and provided adequate bases to determine that the temporary modification did
not involve an unreviewed safety question. Control room drawings were revised
to indicate configuration changes as required by the administrative procedure.
The total number of installed TMs was 27.
The total number of TMs has
remained relatively constant over the past six months.
The majority of the
TMs were not installed on safety-related systems.-
The inspectors concluded
that the TM program implementation by the technical department was good.
4.0
ROOT CAUSE ANALYSES {RCA)
4.1
Incident Report Reviews
Incident reports (IRs) document degradation and anomalous responses of plant
systems and equipment.
They are used by the licensee to investigate and
resolve these and other types of problems.
Procedure NC.NA-AP.ZZ-0006(Q)
(NAP-6), "Incident Report/Reportable Event Program and Quality/Safety Concerns
Reporting System," specifies the requirements for IR initiation,
documentation, and resolution.
IR resolution requires the use of a predefined
root cause methodology.
Specifically, Step 5.2.2 of this procedure requires
the use of one type of root cause methodology, causal factor, and barrier
. .
4
analysis or change analysis for every IR.
The inspectors reviewed selected
open and closed IRs to assess the quality and effectiveness of the licensee's
root cause methodology for both in-process and completed IRs.
The findings
were identified as written below.
4.2
Diesel Room Cooler Safety Auxiliary Cooling System (SACS) Valves
Stick;ng Closed (IR 93-087)
The root cause evaluation conducted following the failure of two emergency
diesel generator (EOG) room cooler valves to open was reviewed to determine
the quality of the root cause determination. A description of this failure,
including the apparent cause and corrective actions were documented in
Licensee Event Report 93-006.
There are 32 SACS supply valves that provide isolation for EDG and emergency
core cooling system (ECCS) pump room coolers.
Each room is provided with two
redundant coolers from independent SACS loops.
The cooling water isolation
valves are Anchor/Darling flex wedge gate valves with Hiller pneumatic
actuators. The Anchor/Darling gate valves used are 3-inch, 4-inch and 6-inch
depending on the cooler design.
The Hiller pneumatic operators use air
pressure to close the valves and springs return the valves to the open
position. The valves are designed to fail open on a loss of air or electrical
power to the solenoid actuator valve.
The valves receive an automatic open
signal following a loss of coolant accident (LOCA) or high room temperature
signal. The valves are in the inservice test program and are stroke time
tested quarterly.
On S~ptember 6, 1993, an operator identified that the "D" EDG room cooler fan
was running, but the SACS valve (EGHV2398D) had failed to open as expected.
The valve was.mechanically agitated and it opened.
The operator then
attempted to open the redundant EDG room cooler SACS valve (EGHV2398H), which
also failed to open.
One SACS EDG room cooler valve in each EDG room was
failed open to assure adequate room cooling, and a RCA was initiated.
The RCA was documented in a memorandum from the system engineer to the
technical manager dated September 30, 1993.
The evaluation stated that a
search of the work order history indicated that 12 of the 32 SACS cooler
valves had failed since 1987, with 2 valves failing twice.
On average, one of
the 32 valves was disassembled every six months.
The failed valves were of
different sizes and were located in several different locations. The RCA
concluded that the gates were binding in the valve body.
The binding was due
to excessive thrust when closing the valves.
The excessive thrust was the
result of a 19~9 design change that replaced the original Crane asbestos valve
packing with Chesterton graphite packing. The Chesterton packing thrust
loading was less than that of the original packing and design packing load.
The reduction in packing load resulted in more thrust being imparted to the
valve stem and disc seating. The additional stem thrust caused the valve disc
to become wedged in the valve seat and the spring force was unable to overcome
the static friction to open the valve.
To reduce the stem thrust, the air
pressure to the Hiller actuator was reduced from 80 psig to 60 psig. This
reduced the closing thrust of the valve by approximately 25% (1000-1500 psig)
5
to compensate for the estimated reduction in packing load thrust (800 psig).
The valves were tested using a Fisher Controls (FlowScanner) to measure the
reduction in unseating thrust (450 psig) required following the reduction in
air pressure.
Changes were also made to the stroke time surveillance duration following the
valve failure. All the SACS cooler valves were stroked weekly. After the
reduction of actuator air pressure, 15 of the 32 valves were put back on the
quarterly surveillance schedule. This test schedule was continued, with no
valve failures, until April 1994 when the valves were placed back on the
quarterly surveillance schedule.
The use of diagnostic test equipment to validate assumptions and the failure
history search were positive attributes of the RCA following the valve
failure.
However, it's not apparent that the root cause techniques described
in NAP 6 were used.
The root cause evaluation information documented on NAP
6, Form NC.NA-AP.ZZ-0006-1,Section IV was not complete.
The failure to use
the RCA techniques narrowed the scope of the potential failure modes
evaluated.
For example, the valve packing issue, that was later identified as
a potential cause following the most recent valve failure in 1994, was not
thoroughly evaluated. The corrective actions implemented following the
completion of the RCA were not successful in preventing repeat valve failures.
4.3
SACS (Hiller) Valves Failed to Operate During OP-15.EG-0102Q (IR 94-185)
On October 22, 1994, two SACS supply valves to the "B" EDG (6-inch valve
1EGHV-2398F) and "D" residual heat removal room coolers (3-inch valve
EGHV-2290H) failed to open during the conduct of the routine quarterly
surveillance test.
The valves opened following mechanical agitation. The
valves were returned to a weekly stroke schedule until corrective actions
could be implemented.
The air pressure supplied to the actuators was checked
and found to be appropriate. A description of the valve failures was
documented in LER 94-017.
A review of the design adequacy of the valve actuators was conducted.
The
valve and actuator vendors assisted the licensee in making this determination.
Calculations were performed to determine the required thrust to unseat or seat
the valves using a disc friction coefficient of 0.5. The margin between
required and available thrust varied between a minimum of approximately 369
psig for the 3-inch valves to approximately 700 psig for the 4-inch and 6-inch
valves. It appears from these calculations that the currently installed
actuators are adequately sized.
A review of the valve packing configuration as a potential cause for the
failure was also conducted.
The original Crane valve packing was replaced
with Chesterton Graphfoil packing in 1988.
In most cases, the rings of Crane
packing were replaced one-for-one with the Chesterton packing.
However, the
control of the number of packing rings installed and the torque values used
6
for the packing gland were not well controlled. This resulted in a variety of
different packing/torque combinations used for the SACS valves. The current
industry practice for graphite packing is to use 4 or 5 packing rings. The
reduction in the number packing rings by half (10 to 5), could reduce the
calculated packing load by approximately 50%.
The RCA stated that industry experience indicates that graphite packing tends
to stick to the valve stem in valves that remain in stationary positions for
long periods of time. Sticking of the packing can substantially increase the
force required to unseat the valves. The sticking can be reduced by improving
the finish on the valve stem.
The root cause evaluation concluded that the valve packing was the most likely
cause of the failure. The corrective actions are to establish configuration
control of the valve packing and repack the valves with the standard 5 rings
of Chesterton packing.
The licensee's cognizant engineers stated that a
thorough inspection of each valve would be conducted to establish a baseline
condition of critical components.
While the corrective actions appear appropriate, the RCA evaluation conducted
for the failure of these valves was not thoroughly documented.
The RCA
consisted of several E-Mail letters that described a chronology of actions
taken to investigate the cause of the failure.
The results of testing and
physical inspection were frequently not included in the RCA documentation .
For example, the information on the number of packing rings and type of
packing removed from the failed valves was not included in the documentation.
Since the conclusion was that packing was the cause of the failure this
information was important to support this conclusion. It was not apparent
that the RCA techniques described in NAP 6 were used.
Nor was it apparent
that a detailed plan to evaluate the failures had been developed.
The
additional documentation to support the conclusion of the RCA was being added
to the RCA at the conclusion of this inspection.
4.4
80° Emergency Diesel Generator Room Cooler Supply Valve 1EGHV2398F
Failed to Stroke Open
On October 29, 1994, during the first weekly stroking of the SACS cooler
valves following the October 22, 1994 failures, one of the diesel room cooler
supply valves failed to open (1EGHV-2398H).
The operators proceeded to fail
the valve open pending an evaluation by engineering.
The investigation of the failure determined that the air pressure being
supplied to the actuator was 85 psig rather than the design pressure of 60
psig. The increase pressure was caused by a faulty air regulator. The
regulator was replaced and the valve was returned to service.
The documented root cause evaluation was not thorough.
The RCA techniques and
forms provided in NAP-6 to conduct the RCA were not used.
The root cause
evaluation documentation consisted of an E-Mail letter that was primarily a
chronology of events.
The cause for the air regulator failure was not
documented.
The fact that a similar valve failed one week earlier that had
the proper air pressure was not discussed.
In addition, prior to 1993 the
- ..
7
SACS cooler valves air pressure was normally set in the 80 psig range, with
infrequent valve failures; therefore, it's not apparent that the additional
air pressure was the cause of this failure. Since the corrective action taken
in 1993 to reduce the air pressure to the actuators had not been completely
s~ccessful, it was not clear that the current reduction would correct this
problem.
The root cause evaluation did not evaluate packing loads or other
potential causes for this failure.
4.5
Conclusions
The RCAs reviewed regarding the failures of the SACS room cooler supply valves
were weak.
The inspectors requested that the licensee provide some recent
examples of thorough RCA.
The quality of the RCAs provided were excellent.
The inspectors concluded that the quality of the RCAs varied considerably and
the standards for RCA were not clearly defined.
The cognizant technical staff
stated that while the quality of RCA documentation was improving, the
rejection rate of RCA documentation was recently as high as 75%.
Based on the
three RCAs reviewed in detail, it did not appear that station supervisors
provided clear expectations or oversight of RCA completed by their staff. In
addition, the guidance provided in the administrative procedures did not.
clearly establish expectations and was often not followed.
The current
administrative procedures required the same level of RCA be performed for
every IR regardless of safety significance or failure history. This approach
appeared to overburden the capability of conducting quality RCA and did not
focus staff efforts on the more safety significant issues. The technical
department staff and management responsible for the RCA program were aware of
the weaknesses in this program.
Efforts were in progress to make significant
revisions to the program to improve the overall quality of RCA and corrective
action program.
5.0
PLANT MODIFICATIONS
5.1
Reactor Protection System (RPS) Alternate Power Supply Transformer
Replacement (Hope Creek 4EC-0032)
This modification involved the replacement of the RPS alternate power supply
transformers (1AX432, 1BX432) with ferroresonant voltage regulation
transformers.
The RPS alternate power supply is mainly used for maintenance
purposes.
The reason for the change was that the two series connected RPS electrical
protection assemblies (EPAs) were tripping on undervoltage when large pump
motors or combinations of motors were started. Each EPA contains a breaker
that is controlled by solid-state electronics. They provide protection for
the RPS loads against overvoltage, undervoltage, and underfrequency.
The design intent of the modification was to reduce the voltage dips caused by
the starting of the large motor loads on the input side of the transformers,
and hence any unnecessary trips caused by the EPA's .
. '
8
Rev;ew - Unresolved Item 50-354/95-02-01
The inspectors reviewed for:
(1) the performance changes in the voltage
regulation; and (2) the suitability of voltage waveform harmonic distortion
with EPA calibration and RPS loads.
The licensee performed tests to determine the voltage levels under conditions
of high and low input voltage under load conditions.
The inspectors reviewed
the test data for the starting of fan motors {lAVH-404 lBVH-105.)
The data
showed good voltage regulation, but the waveform was quite distorted.
The inspector asked what the total harmonic distortion (THO) specification was
for the RPS bus.
The licensee stated that there was no THO specification for
the RPS bus in the documentation on site, and would contact the NSSS vendor
for confirmation.
The vendor stated that the MG sets, which supply power to
the RPS buses, did specify an allowance for 5% THO in the purchase
specifications. The vendor added, however, that there was no design
requirement, for the Hope Creek plant, in the RPS design specifications as it
relates to the system, its components, or the power buses.
The licensee said that the THO on the RPS busses would be measured when plant
conditions permit.
The determination of the THO on the RPS buses when
connected to the alternate power supply and the suitability of the measured
THO value for the connected loads is an unresolved item.
The licensee plans
to measure the THO on the bus, while utilizing the alternate supply, and will
determine the suitability for the RPS loads.
{Unresolved Item
50-354/95-02-01)
The inspectors reviewed the design analysis for the calibration of the EPAs
when used with the distorted waveform.
The licensee had information from
another utility that had used ferroresonant transformers for this application.
The information was that the EPAs compute the average value of the rectified
waveform, not the root-mean-square (RMS) value of the waveform.
The
requirements for overvoltage and undervoltage protection for the RPS bus loads
are given in terms of RMS voltage for a sine wave, not the average value
voltage for a sine wave.
The licensee recognized that a different method of calibrating the EPAs had to
be implemented for the case using a harmonically distorted waveform.
They saw
from plant testing, during the design stages for the modification, that there
was no noticeable change in the shape of the ferroresonant transformer voltage
waveform throughout the load range of 0 to 70 amps.
They conducted tests to
determine the RMS value of the distorted wave at which the EPAs tripped. They
knew where the overvoltage and undervoltage setpoints were set for the RMS
sine wave case from calibration data.
The licensee then used a graphical method and mathematical spreadsheet formula
to determine the average and RMS value of the distorted wave.
The calculated
values correlated correctly with the test data. Then they calculated the
amplitude ratio of the distorted wave case to the sine wave case.
. .
9
The amplitude ratio was used to calculate and set the RMS sine wave
calibration of the EPAs for the distorted waveform case. This allowed the
licensee to use a standardized EPA calibration method, but account for the
distorted waveform case by setting the trip points to values determined by a
formula.
The inspectors reviewed the derivation of the amplitude factors and the
resulting overvoltage and undervoltage set points for the EPAs.
The graphical
method used in the derivation considered enough points of the waveform to be
valid. The design analysis was clear, showed reasons for the options chosen,
and was reasonable.
The inspectors walked down the installation of the ferroresonant transformers
in the MG set room and found the material condition excellent.
Conclusion
The modification appropriately considered the influence of harmonic distortion
on the calibration of the EPAs for the alternate RPS supply.
The engineering
design analysis of the waveforms for calibration standardization was
noteworthy.
The suitability of the RPS loads to possible change in total harmonic
distortion caused by the ferroresonant transformer was not determined and is
an unresolved item.
The licensee plans to measure the THO on the bus, while
utilizing the alternate supply, and will determine the suitability for the RPS
loads.
(Unresolved Item 50-354/95-02-01)
5.2
Overfill Protection System (OPS) Logic Correction (Hope Creek 4HE-163)
This modification was a result of an unexpected trip of the main turbine
generator with consequent reactor trip. This event occurred on
October 2, 1994.
The cause of the trip was traced to a design deficiency in
the digital feedwater control system (DFCS).
The design requirement was to require two series contacts for high reactor
vessel water level to cause a main turbine trip. The system was actually
configured such that the contacts were wired in parallel.
The reactor vessel water high water level for OPS is sensed in three channels,
which are separate from the feedwater level channels. These OPS channels are
combined to produce a normally energized two-out-of-three logic in two
separate trains.
Each train drives one contact. The train contacts should
have been connected in series to the energize-to-trip turbine trip relay. The
licensee believed that one of the train contacts was intermittent, but was
unable to confirm this with follow-up testing.
Review
The inspectors reviewed the modification package and verified that the wiring
diagrams actually showed the two sets of contacts in series.
10
The inspectors reviewed the licensee design review of the DFCS, dated
November 28, 1994. Three root causes were determined:
(1) the schematic was
not available during the development of the wiring diagrams, installation
instructions, or testing requirements, so that correctness could not be
determined; (2) the design process had many elements in parallel, so that the
flow from requirements to the schematic and to the wiring diagrams was
uncoordinated; and (3) the design outputs were not reviewed for correctness
with the design analysis.
The inspectors interviewed the design engineer and determined that the root
causes were valid. The design analysis was performed at the same time that
the wiring diagrams were being developed. The design engineer noted the
problem with the OPS output to the turbine trip circuit.
He verbally informed
the proper people of the necessity for a design change.
The proper action did
not take place, and the engineer was preoccupied with other issues, so he
forgot to follow up.
A very strong factor that negatively influenced the
design cycle was aggressive scheduling. This was typified by the fact that
some of the plant level power tests were not done as originally scheduled.
Conclusion
The licensee's review of the design process accurately depicted the factors
and actions that caused this small, readily understood, flawed design detail
to develop into a challenge of the plant safety systems.
The root causes were
well documented, reasonable, detailed, and valid.
6.0
QUALITY ASSURANCE (QA) AUDIT REPORTS
The inspectors reviewed two audit reports and one surveillance report
conducted by the licensee's QA organization. The reports were assessments of
Hope Creek technical department activities. The reports were reviewed by the
inspectors to assess the quality of the independent oversight provided by the
QA organization.
The two QA audit reports reviewed were 94-133, "Technical Support Nuclear
Department,
11 and 94-133-3, "Reactor Engineering & Nuclear Fuel Design.
11
A
Quality Assurance surveillance 94-0315, "Control of Temporary Modifications,
11
was* also reviewed.
The inspectors concluded that the QA oversight of the technical department was
good.
This conclusion was based on the scope and number of recent audits and
surveillances conducted of technical department activities. However, the
inspectors observed that the summary section of the audit reports described
specific technical issues and did not provide an overall assessment of the
issues or the activities of organization audited.
An overall assessment as to
the quality of the activities or organization reviewed would be useful to
assist plant management in assessing performance .
11
7.0
POTENTIAL FOR LOSS OF AUTOMATIC ENGINEERED SAFETY EQUIPMENT ACTUATION
SIGNAL
The licensee was alerted to a potential loss of automatic engineered safety
(ESF) equipment actuation signal under certain circumstances by a phone call
from another utility that had a solid state protection system (SSPS) similar
to the Salem plants. The design engineers in Engineering and Plant Betterment
(E&PB) immediately researched the design basis drawings to see if a similar
problem did in fact exist at the Salem plants. They discovered that a
postulated seismic event or steam line break in the turbine building could
render both trains of SSPS inoperable or susceptible to a single failure.
Either one of the postulated design basis events could potentially lead to the
loss of ESF actuation signal because of certain wiring and fusing
configurations that existed.
The wiring for the non-Class IE turbine stop valve limit switch contacts, the
auto-stop oil relay contacts, and the reactor coolant pump (RCP) breaker open
contacts pass through termination boxes which are located in the turbine
building. The turbine building is a non-seismic structure and is susceptible
to the harsh environment that could be caused by a steam line break.
The scenario assumed a postulated failure that could cause a short to ground
of the non-Class IE wiring that would consequently cause loss of the 15 and 48
Vdc power supplies in one or both trains. The resulting impact could be
either a partial or total loss of the ESF actuation function of the SSPS, but
the reactor would trip.
NRC Information Notice 95-10 was issued to alert licensees to the to the
potential for loss of the ESF actuation function according to the above
scenarios.
The licensee reviewed the Hope Creek design basis (Letter ELE-95-0031,
February 3, 1995), and found that the plant did not have the same
vulnerability because the individual circuits were fused as well as interposed
with relays prior to going to the RPS trip circuits.
7.1
Solid-State Protection System (SSPS) Modification (lEC-3403, 2EC-3351)
The licensee developed a modification for the Salem SSPS to reconfigure the
input bay wiring such that the logic power supplies will not be affected by a
postulated short to ground of the input wiring that comes from contacts in the
turbine building.
The licensee declared both trains of the SSPS inoperable
for Salem 1 and 2.
They applied for and received NRC enforcement discretion
to implement the modification.
The new configuration moved the 120 Vac power feed for the logic power
supplies, which are individually fused, upstream of the existing 15 amp fuses.
This left the fuse in a position to provide the proper protection for the
input bay wiring.
In addition, the 15 amp fuses were replaced with 5 amp
fuses that coordinated properly with the upstream breaker.
' '
12
The licensee stated that they performed a modification pre~installation
rehearsal on identical equipment not housed in the plant. This allowed them
to refine the installation procedure without the risk of a plant trip.
SSPS Power Supply Failures
During the implementation of the design change that addressed isolation of
non-safety inputs from safety-related power, the licensee experienced some
SSPS power supply (PS) problems in Salem 1. The NRC enforcement discretion
was rescinded because the power supply problems changed the conditions under
which the discretion was granted.
The licensee shut down Salem 1 and 2 in
accordance with the Technical Specifications.
There are two independent power supply chassis for each train.
Each chassis
and contains one 15 Vdc section and one 48 Vdc section.
Each chassis has a
model number, which indicates the incorporation of certain vendor internal
power supply design changes.
Each section uses a switching regulation
technique and has overvoltage and overcurrent protection. The respective
sections from each power supply are diode coupled to form a single 15 Vdc
supply and a single 48 Vdc supply for each train.
The initial SSPS power supply problem was found during the power-down sequence
prior to actually starting the modification.
When the breaker for train A,
PSI 15 volt section, was manually tripped, the breaker for PS2 15 volt section
would trip.
In the course of troubleshooting this malfunction, the licensee
became aware of other anomalies in the power supplies and decided to bench
test all the SSPS power supplies for Salem 1 and 2.
The wiring and fuse
changes for the modification proceeded in parallel with the power supply bench
testing.
The four power supplies (three model 101, one model 100) for the SSPS of Salem
1 were removed, bench tested in the I&C shop, and did not meet specifications.
The four power supplies (two model 103, one model 101, one model 100) from
Salem 2 SSPS were tested and all but one (model 101) were acceptable.
The
three acceptable Salem 2 power supplies were then installed in Salem 1 SSPS.
One power supply (model 103), that was a spare from another utility, was also
bench tested and installed in Salem 1.
The SSPS for Unit 1 was then returned
to service, the post-modification testing was performed and passed.
One spare power supply from the warehouse (model 100) was bench tested,
installed in Salem 2, but failed, and was removed from service. Three spare
power supplies (all model 103) from other utilities, along with one spare
(model 103) from the PSE&G warehouse, were bench tested and installed in Salem
2.
The SSPS for Unit 2 was then returned to service and the post-modification
testing was performed and passed .
c
13
Rev;ew
The inspectors reviewed the design change package (DCP), "SSPS Interface to
Turbine Building," for wiring changes, fuse coo rd i nation, fuse commercial
dedication, fuse seismic qualification, and post-modification testing, and
found them adequate.
The post-modification test procedures for the SSPS were
based on the same as used in the plant functional test.
The inspectors determined by interviews that the output supply volta~es are
verified on a regular basis, but the output ripple voltage is not measured or
verified.
The inspectors audited the testing of the modification, the bench testing, and
the troubleshooting of the power supplies. The Salem plant manager requested
assistance from E&PB management, and engineering specialists were dispatched
around-the-clock to provide in-depth technical analysis for the
troubleshooting tasks. Plans were developed for the in-situ testing of the
supplies. Plans were developed for inspection and bench testing of the
removed power supplies. The power supplies were visually inspected and
cleaned.
The bench test covered overvoltage function, overcurrent function,
output ripple measurements, and output voltage regulation. Test data was
recorded and compared to vendor specifications for acceptance. Associated
system indications, consequent equipment conditions, and annunciators were
examined and probable causes documented by the Salem system engineers .
The original problem was found to be a single component failure in Salem 1,
train A, PS2 15Vdc section. A short in the regulator circuit was caused by a
wire being grounded to a heat sink, which most likely degraded the regulation
of the PS2 15 volt section. The presumed scenario was that when the redundant
supply was tripped, the increased load on PS2 caused overshoot of the output
due to degraded regulation, and the overvoltage protection circuit tripped the
supply.
The troubleshooting showed that several probable causes were responsible for
five of the original eight and one PSE&G spare power supplies specification
malfunctions. Table 1 shows the distribution of the malfunction categories.
The malfunctions can be categorized as random component failures stemming from
latent manufacturing defects and age degradation of electrolytic filter
capacitors.
No malfunctions were noted in the model 103 series power
supplies. The theory was that they were manufactured later than the other
series, had internal design changes incorporated, and had newer electrolytic
capacitors. The six supplies found out of specifications were in the I&C
shop, tagged as defective .
'
L
14
TABLE 1: NUMBER OF POWER SUPPLY MALFUNCTIONS BY TYPE AND MODEL
PERFORMANCE
Model 100
Model 101
Model 103
AREA & PS
SECTION
High Ripple
(note 1)
48 Vdc
1
2
15 Vdc
2
Regulation
48 Vdc
1
2 (note 2)
15 Vdc
1 (note 4)
2 {note 3)
TABLE 1 NOTES:
(1)
Suspect aged electrolytic capacitors.
(2)
One section, transistor shorted/overheating.
(3)
One section, wire was shorted to heat sink.
One section, suspect faulty electrolytic capacitors.
(4)
Unauthorized use of metallic standoffs caused regulator
failure by creating a shorting path. Spare from PSE&G
warehouse.
Through the visual inspection and troubleshooting tasks, the licensee
uncovered some design issues that were resolved as described below.
1.
The general warning alarm for the SSPS, that provides local and control
room warning, was discovered to not alarm if the power supply voltages
levels were degraded, but not completely lost. This meant that if a
supply failed and the redundant supply had an undetected failure of the
output regulation, then an alarm would not happen.
The vendor stated
that the general alarm was only designed to detect a complete loss of
voltage, not a degraded voltage condition. This could lead to failure
within a train, but the redundant train would be available, and the SSPS
would still meet the design basis. The licensee stated that they will
pursue a design change with the vendor to improve the detection
capability of the general warning alarm.
2.
The design problem discovered by another utility was confirmed. A
resistor in the overvoltage protection circuitry of the 48 Vdc section
of a supply was the incorrect wattage rating in the 100 and 101 models.
The resistor could dissipate up to twice the rated wattage.
If failure
3.
15
of the resistor occurred, the vendor stated that the overvoltage
setpoint could drift up to 10%.
For the one model 100 supply installed
in Salem 1, the licensee checked the resistor for proper value after
bench testing. The particular model 100 supply was previously installed
in Salem 2 SSPS since plant start-up, and the resistor had not failed or
caused output voltage problems.
The E&PB engineering staff has
recommended that the resistor be replaced at the next outage to the
value and wattage as recommended by the vendor.
The licensee noticed that the output breaker identification markings
were different in the supplies. The identification difference concerned
a time delay feature of the breakers. The licensee contacted the
vendors involved and the breaker suppliers to determine if the breakers
were identical. The review of the information indicated that the
breakers were all of the correct type.
At the end of the inspection, the licensee had not yet finished the formal
root cause analysis of the SSPS power supply failures.
The E&PB engineering
staff stated that preventive maintenance procedures to measure the output
voltage ripple, measure the output voltage regulation, and replace the
electrolytic filter capacitors on.some age-related basis would be examined.
7.2
Conclusions
The E&PB engineering staff showed appropriate defense of the design basis by
their research of the actual Salem design status when information was given to
them from another utility. The engineers received the information verbally
and acted in a timely manner.
Management acted promptly on the information
presented to them by the engineers. This was an example of an appropriate
safety-conscious decision process.
Management involvement and direction was evident by the quick dispatch of
engineering specialists on a full 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> day coverage for troubleshooting,
analysis, and corrective action when the SSPS power supply problems occurred.
The troubleshooting work process was planned, controlled, and documented.
The
analysis of failures was accurate, detailed, documented, and technically
sound.
The Salem system engineers provided accurate analysis in the
operability determinations of the SSPS.
The design issues were put in the perspecti~e of plant safety. There was
evidence of effective communication with the engineering vendors and suppliers
involved.
8.0
EXIT MEETING
An exit meeting was held on February 17, 1995, with members of the licensee's
staff noted in Attachment 1.
The inspectors discussed the scope and findings
of the inspection. The licensee had no disagreements with the findings.
The
inspectors received and reviewed proprietary material during the inspection
and used the material only for technical reference.
No part of the material
was knowingly disclosed in this inspection report.
I - *"
ATTACHMENT 1
EXIT MEETING ATTENDEES
Public Service Electric and Gas
P. Bernini, Principal Engineer - QA Programs
J. Benjamin, Director QA/NSR
M. Bursztein, Nuclear Electrical Engineering Manager
J. Clancy, Technical Manager - Hope Creek
J. Defebo, Hope Creek - Quality Assessment
W. Denardi, Sr. Projects Engineer
B. Diaz, Hope Creek Projects
G. Englert, Civil Structural and Programs Manager
S. La Bruna, Vice President Nuclear Engineering
C. Manges, Station Licensing Engineer - Hope Creek
C. Nentwig, Nuclear Engineer Design
J. Priest, Engineer Licensing and Regulation
D. Smith, Principal Engineer Nuclear Licensing
F. Thomson, Manager Licensing and Regulation
U. S. Nuclear Regulatory Commission
R. Summers, Senior Resident Inspector - Hope Creek