ML18101B205
| ML18101B205 | |
| Person / Time | |
|---|---|
| Site: | Salem  |
| Issue date: | 01/23/1996 |
| From: | Calvert J, Ruland W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML18101B203 | List: |
| References | |
| 50-272-95-81, 50-311-95-81, NUDOCS 9602080027 | |
| Download: ML18101B205 (27) | |
See also: IR 05000272/1995081
Text
./ **
DOCKET/REPORT NOS.
LICENSEE:
FACILITY:
DATES:
INSPECTORS:
LEAD INSPECTOR:
APPROVED BY:
9602080027 960131 ..
ADOCK OS000272
G
U. S. NUCLEAR REGULATORY COMMISSION
REGION I
50-272/95-81
50-311/95-81
Public Service Electric and Gas Company
Hancocks Bridge, New Jersey 08038
Salem Nuclear Generating Station
October 5, 1995 - November 13, 1995
Lonny L. Eckert, Radiation Specialist
Jenifer M. Shannon, Electrical Engineer
n . Ca vert, Reactor ngineer
Electrical Engineering Branch
Division of Reactor Safety
LJ~~-1'1-~
Willfam H. Ruland, Chie
Electrical Engineering Branch
Division of Reactor Safety
01;/~/9--&
Date
.-
TABLE OF CONTENTS
Page Nos.
EXECUTIVE SUMMARY .
1.0
INTRODUCTION
. . . . . . . . . . .
1.1
Special Inspection (IP 92903) .
1.2
Significant Event Response Team
1.3
Overhead Annunciator System ..
1.4
Event Sequence ....... .
. . .
(SERT)
2.0
EVALUATION OF EMERGENCY PREPAREDNESS ....
2.1
Emergency Assessment and Classification ...... .
2.1.1 Evaluation of Event Recognition and Declaration .
.
2.1.2 Evaluation of the Initial Notification of the Loss of
Annunciator Event .............. .
2.1.3 Evaluation of the Appropriateness of the Alert
Declaration ................... .
2.2
Emergency Action Level Scheme ............. .
2.2.1 Appropriateness of the Loss of Annunciator EAL
2.2.2 Discussion of the EAL Scheme With the States of New
Jersey and Delaware ............ .
2.3
Emergency Response Organization (ERO) Augmentation ..
2.3.1 Emergency Operations Facility (EOF) ....... .
2.3.2 Operations Support Center (OSC) ......... .
2.3.3 Technical Support Center (TSC) .......... .
2.3.4 Licensee Emergency Response Organization Notification
Process ..................... .
2.3.5 Emergency Response Organization (ERO) Augmentation
Summary . . . . . . . . . . . . . . . . . . . . . .
2.4
Communications/Reporting/Notifications ........ .
2.4.1 Notification and Reporting of Events ...... .
2.4.2 Communications with Off-Site Response Organizations
3.0
SIGNIFICANT EVENT RESPONSE TEAM (SERT) PERFORMANCE
3.1
Emergency Preparedness Review ...... .
3.2
Human Performance and Procedural Review .. .
3.3
Integration and Usability of Operational Procedures
3.4
Overall SERT ...... .
4.0
TECHNICAL ROOT CAUSE PERFORMANCE
4.1
Causal Factors Analysis ..
4.2
Root Cause ....... .
4.3
Operability Determination and Compensatory Periodic Tests 4.4
Comparison With the 1992 Salem Unit 2 OHA Event
4.5
Generic Implications and Notifications
5.0
MANAGEMENT OVERSIGHT
6.0
OVERALL CONCLUSIONS .
7.0
EXIT MEETING AND TELEPHONE CALLS
ii
i i i
1
1
1
2
3
4
4
4
6
6
7
7
8
10
10
10
10
11
11
12
12
12
13
13
14
14
16
16
16
17
17
18
19
19
20
21
..
EXECUTIVE SUMMARY
Salem Inspection Reports 50-272/95-81; 50-311/95-81
October 5, 1995 - November 13, 1q95
This special inspection evaluated the performance of PSE&G in response to a
30 minute undetected loss of the Salem Unit 1 overhead annunciator (OHA)
system.
The inspection methods involved independent oversight of licensee
activities associated with this event, walkdown of control room OHA equipment,
and interviews with some of the personnel involved in the event.
The
inspectors also examined PSE&G's significant event response team (SERT) and
root cause investigation reports.
The event occurred on October 4, 1995, at 10:35 p.m., when Salem Unit 1
experienced a failure of the OHA system that was not indicated to the
operators.
The operating crew recognized the failure approximately 30 minutes
into the event, and reset the system 30 minutes after recognition.
The reset
was not entirely successful in that certain anomalies were still observed with
the system.
An alert was declared on October 5, 1995, at 1:38 a.m.
Following
another OHA system reset and diagnostic testing performed by the system
engineer that indicated proper functioning, the alert was terminated at
5:22 a.m.
Before, during, and after the event, Salem Unit 1 was shutdown and defueled.
Salem Unit 2 was in cold shutdown.
The OHA systems are identical for Unit 1
and Unit 2.
There were no turnovers in the operating crew during the event.
In the emergency preparedness (EP) area, the inspectors found that although
the conditions for an alert declaration had been exceeded and identified by
the operating crew, the operating crew decided not to declare an alert. The
operating crew initially dispositioned the event as a I-hour report to the
NRC, but the report was not initiated in a timely manner.
The event was
eventually classified as an alert at 1:38 a.m., approximately ~h hours after
the OHA system was observed to be inoperable.
(TSC) activation was not timely, and PSE&G did not meet their emergency plan
staffing requirements for the TSC.
After the event, PSE&G appropriately
revised the emergency action levels (EALs) covering loss of annunciator
events, but failed to discuss and seek agreement with the States prior to the
implementation of the revised EALs.
Three violations based on these emergency
preparedness findings were issued.
Also in the EP area, the inspectors noted that the operations support center
(OSC) activation was timely.
The emergency operations facility (EOF) manning
was a conservative action on the part of PSE&G and was within the level of
discretion provided by their emergency plan.
The licensee's emergency
response organization (ERO) call-out process was weak as shown by, for
example, the fact that a duty call list was not maintained, and a significant
number of those people that were called failed to respond or questioned the
need to respond.
The inspectors also noted that with the exception of the
State of New Jersey, representatives from offsite response organizations found
ii i
- -*
the communications provided by the licensee to be commendable.
PSE&G's use of
the phrase, "5-minute alert," hindered the understanding of the State of
New Jersey Bureau of Nuclear Engineering (BNE) regarding the event.
Iri the emergency respons1~ equipment area, the inspectc rs found that there was
no indication to operators of major OHA system problems that would prevent
alarm processing. This type of undetected failure also occurred in the
December 1992 Salem Unit 2 OHA event, and therefore this is the second
occurrence of this type of "silent" malfunction without failover to the
backup. This made the equipment unavailable to perform the OHA function, and
did not provide the operators with sufficient information to determine that
unavailability.
One violation based on the above finding was issued.
The inspectors noted that PSE&G's determination of the most likely root cause
was technically sound and was the best fit with the actual indications of
failure. Operator errors in keystrokes and interface switch settings were not
a factor in this event due to effective corrective actions for these areas
taken after the 1992 Salem Unit 2 OHA event.
The OHA operability
determination and the compensatory tests were sufficient for use in the
respective plant modes.
The inspectors observed an overall weakness in the effectiveness of the
engineering, operations, and training organizations to support the plant
operators with a unified set of OHA knowledge, skills, and abilities to
recognize failure indications, to determine operability, and to take proper
corrective action.
In the management oversight area. the inspectors concluded that management was
actively involved in the alert declaration and overall direction of the
failure analysis process .
iv
- *.~.. _,
~
DETAILS
1.0
INTRODUCTION
1.1
Special Inspection (IP 92903)
On October 4, 1995, at 10:35 p.m., Salem Unit 1 experienced an unannunciated
failure of the Overhead Annunciator (OHA) system.
The operating crew
recognized the failure approximately 30 minutes into the event.
The operators
took action to reset the system in another 30 minutes after discovery of the
failure. The reset was not entirely successful in that certain anomalies were
still observed with the system.
An alert was declared on October 5, 1995, at
1:38 a.m.
Following another OHA system reset and diagnostic testing performed
by the system engineer that indicated proper functioning, the alert was
terminated at 5:22 a.m.
Before, during, and after the event, Unit 1 was shutdown and defueled.
The
OHA systems are identical for Unit 1 and Unit 2.
There were no turnovers in
the operating crew during the event.
Following the event, NRC managers determined that a special inspection was
warranted to gather event-related information. A charter was formulated for
the special inspection (Attachment 1). The inspectors began their inspection
on October 5, 1995.
The NRC had previously inspected a Salem Unit 2 OHA event during an Augmented
Inspection Team (AIT) inspection in December 1992, Report Number 50-272/92-81,
50-311/92-81.
The inspection methodology involved independent oversight of licensee
activities associated with this event. The inspectors conducted a detailed
walkdown of the control room and overhead annunciator equipment.
Independent
interviews were conducted with some of the personnel involved in the event, as
well as training and procedure-writing personnel.
The inspectors used
techniques set forth in NUREG/CR-5455, "Human Performance Investigation
Process," for guidance in causal factors identification. The assessments of
the licensee's reports by the inspectors were based upon the findings
described in the SERT and root cause investigation team reports.
The
inspectors supplemented the document review with observation of the activities
of the various teams and interviews of team members.
The licensee reports had
not gone through licensee corrective action review procedures at the time the
inspectors reviewed the reports, so the recommended corrective actions of the
SERT are not covered in this inspection report.
The inspectors compared the
final versions of the reports to the versions used during the inspection and
determined that there were no substantive revisions that-would change the- -
assessments.
1.2
Significant Event Response Team (SERT)
The General Manager-Salem Operations chartered a SERT because the loss of OHA
event involved several significant hardware and personnel performance issues.
The SERT charter required investigation of the.following issues: the failure
mode of the OHA system; the appropriateness of the alert classification; and
the performance of the emergency response organization.
~-
') .
. *.~ . -.:.,;
.
2
The SERT was tasked to investigate the event in accordance with Salem Station
Procedure NC.NA-AP.ZZ-006l(Q), Rev. 6.
The team was composed of
representatives from operations, maintenance, system engineering, emergency
preparedness, independent oversight, and nuclear safety review.
The SERT had
been tasked to determine the sequence of events, determine the root causes,
and provide recommendations for corrective actions.
The SERT consisted of two subteams.
One subteam examined the technical root
cause, while the other subteam retained the SERT title and examined the human
performance and emergency response issues. The final SERT report contained
the results of the two subteam findings.
1.3
Overhead Annunciator System
The OHA system is a high speed distributed data acquisition and display
system.
The purpose of the system is to detect alarm conditions and indicate
these conditions to control room operators. The alarm conditions
are provided to the operator through overhead window lights with audible
alarms, a CRT screen, and a printer.
The system has two sequential event recorders (SER) that receive alarm
information, process .the information, and send it to distributed logic
controllers. Outputs from the distributed logic controllers drive the
overhead windows and alarms.
The two SERs are designed to act as a primary
and hot standby failover pair. The design intent was that when the primary
SER, fails, the hot standby SER takes over control. This failover mechanism
is not active as long as SER-A periodically resets a watchdog timer circuit.
If SER-A fails to reset the watchdog timer, the circuitry is designed to
automatically switches data flow through SER-B.
The CRT is located on the control console and displays all alarm and reset
events along with the current time.
The control console also contains push
buttons that the operator uses to silence, acknowledge and reset the overhead
alarms.
Various internal failures are identified and annunciated by the OHA system.
These alarms are combined into an independent control console group alarm.
The operator panel is installed in the equipment cabinets adjacent to the -
control room.
The panel consists of pushbuttons for obtaining printed reports
containing information related to system status.
The Remote Configuration Workstation (RCWS) is a personal computer that
provides a means to configure the system and is located -in the same area -as -
the operator panel. Access to the keyboard for the RCWS is by administrative
keylock control .
...~ ... -... -
-
~**
., -, . ~-
.:,.: - ~ .. - ; .... *-*;;.;
..
3
The licensee took corrective actions after the 1992 Salem Unit 2 OHA event as
a result of their root cause investigation. These equipment corrective
actions were installed in the OHA of both units and concerned mainly key
access to the computer keyboard, electrical noise susceptibility reduction,
operator initiated independent built-in test of function, and indication of
test failure to operators.
1.4
Event Sequence
Some details of the event are described here to aid in the correlation of the
other sections of the report.
On October 4, 1995, at 10:22 p.m., the breaker for service water bay sump pump
.. "11" was opened as part of a routine surveillance (typically performed
nightly).
OHA system window A-41, "auxiliary alarm system printer," and the
auxiliary alarm system typewriter actuated as expected.
At 10:35 p.m., OHA
window A-21 actuated; this was the last known alarm to have been properly
processed and displayed by the OHA system.
At 11:03 p.m., the "11" service water sump was reported to the control room as
being full, but OHA window B-29, "11-13 service water pump sump area level
high," had not actuated. This was the first indication to the operators that
there was a failure of the OHA system.
The sump pump breaker was closed with
proper acknowledgment from the auxiliary alarm typewriter.
However, OHA
window A-41 failed to annunciate.
At this time, the Nuclear Control Operators
(NCOs) recognized that there was a problem with the OHA system.
At 11:05 p.m., the NCOs initiated an operability check by implementing OHA
system normal Operating Procedure Sl.OP-SO.ANN-OOOl(Q).
system valve 1SS104 was cycled with proper acknowledgment from the auxiliary
alarm typewriter.
Once again, OHA window A-41 failed to annunciate. Also, a
lamp check was performed locally on the 104 panel by a Nuclear Equipment
Operator (NEO) with no response from OHA window C-9.
At 11:12 p.m., an OHA lamp and hourly test was performed.
All OHA windows
illuminated and the group alarms worked.
However, no messages of successful
tests were received on the OHA cathode ray tube (CRT) screen.
OHA window A-9,
"annunciator trouble alarm," did not alarm and clear. Also, the "OHA Trouble"
console alarm was not observed to have alarmed.
It was at this point that the
Senior Nuclear Shift Supervisor (SNSS) determined that the OHA system was
At 11:30 p.m., in accordance with "Loss of Overhead Annunciator System,"
Procedure Sl.OP-AB.ANN-0001, the operators reset. SER-A .. After this res.et and
system testing, the operators were confident that the OHA was functioning
properly, but the reset was not entirely successful in that certain anomalies
were still observed with the system.
An alert was declared on
October 5, 1995, at 1:38 a.m.
The system engineer reset the system again and performed diagnostic testing to
show that the OHA system was functioning properly.
The alert was terminated
at 5:22 a.m.
4
2.0
EVALUATION OF EMERGENCY PREPAREDNESS
2.1
Emergency Assessment and Classification
2.1.l Evaluation of Event Recognition and Declaration
The loss of annunciators was not declared as an alert by the SNSS despite
having sufficient information by about 11:12 p.m., to have declared the event.
The failure was identified by licensee management and the alert was
subsequently declared at 1:38 a.m., on October 5, 1995, after management
review of the situation (see Section 2.1.3 for NRC's conclusion on the
appropriateness of the action of declaring an alert at the time it was made).
During the inspectors' interviews with the operating crew, who are senior
reactor operator (SRO) qualified, the inspectors found that both individuals
had reservations about declaring the alert due to its minimal safety
significance. The SROs believed that not declaring the alert was appropriate
because mobilizing large numbers of emergency response personnel would have
posed more of a risk, and that the attention that would have been aroused as a
result of the alert declaration was unwarranted.
The inspectors concluded
that these statements suggest confusion on the part of the on-duty operating
crew with regard to the responsibilities of the licensee and the States
(New Jersey and Delaware).
The inspectors determined that the following factors contributed to the
operating crew's failure to declare an alert in a timely manner.
The Salem Emergency Classification Guide (ECG) was confusing/incomplete
as the SNSS had considered the ECG to be a guide.
The loss of annunciators when in mode 6 (defueled) does not merit an
alert declaration based on the latest guidance by the Nuclear Management
and Resources Council, Inc., NUMARC/NESP-0007, "Methodology for
Development of Emergency Action levels." The NUMARC-NESP/0007 document
has received generic acceptance by NRC in NRC Regulatory Guide 1.101,
Revision 3.
The NUMARC EAL guidance establishes mode specificity to the
EALs which address a loss of annunciator event. Also, per NUMARC/NESP-
0007, loss of annunciators is not an alert unless a transient is in
progress at the same time.
However, this guidance had not yet been
approved for Salem at the time of the event.
The ECG was not used properly because it had not been stressed
adequately that if an EAL initiating criteria is satisfied, the event is
intended to be classified in accordance with the ECG even if the
Emergency Coordinator does not agree with the classification level .
.-
5
The ECG, "Introduction and References" section, Step V.A. requires, in
part, that "The ECG is a guide.
The EALs described in the ECG are not
all inclusive and will not identify each and every cor~'tion, parameter
or event that could lead to an event classification. If the Emergency
Coordinator, using his best judgment, determines an Initiating Condition
has been satisfied but the specific EAL is in question, he/she should
promptly classify the event in accordance with the Initiating Condition.
In any event, if the plant conditions are equivalent to one of the four
emergency classes ... , that classification should be declared."
The ECG was not used because the operating crew had been trained on a
complete revision of the Emergency Action Level (EAL) scheme in which
loss of annunciators when-in mode 6 was not an alert.
The training for the ECG was less than adequate as there were no
terminal learning objectives for the SNSS, Nuclear Shift Supervisor
(NSS), and the Nuclear Shift Technical Advisor (NSTA) in regards to
demonstrating an understanding of licensee and state capabilities and
responsibilities.
Also, the Emergency Preparedness and Radiological Support Manager
informed the inspectors that the licensee did not use watch-standing
senior reactor operators in activities with personnel from off-site
response organizations. Such activities *such as EAL training or
technical liaisons could foster a more complete understanding of roles
and responsibilities of the various response organizations.
The Salem ECG contains EALs and initiating conditions for event
classifications. The ECG, Section lOB, "Loss of Instrumentation/
Annunciation/Communications," required an alert declaration if "Loss of most
or all (>75%) overhead annunciators (excluding a scheduled test or maintenance
activity for which pre-planned compensatory measures have been implemented)
AND 15 minutes have elapsed since the loss of annunciators."
The ECG, "Introduction and References" section, Step V.A. requires, in part,
that "If the Emergency Coordinator, using his best judgment, determines an
Initiating Condition has been satisfied but the specific EAL is in question,
he/she should promptly classify the event in accordance with the Initiating
Condition.
In any event, if the plant conditions are equivalent to one of the
four emergency classes ... , that classification should be declared."
As noted previously, the loss of annunciators was not promptly declared as an
alert_by the SNSS despite having recognized that the EAL initiating criteria
had been satisfied at about 11:12 *p.m.
The alert was subsequently declared at
1:38 a.m., by the SNSS after management had reviewed and discussed the event
with the SNSS (detailed further in Section 2.1.3) .
6
10 CFR 50.54(q) requires, in part, that "A licensee authorized to possess and
operate a nuclear power reactor shall follow and maintain in effect emergency
plans ~.hi~h meet the standards in 50.47(b) and the requirements in Appendix E
of this part." Therefore, the failure to promptly declare the loss of
annunciator event as an alert, in accordance with the licensee's procedures,
was assessed as a violation of 10 CFR 50.54(q) (VIO 50-272/95-81-01).
2.1.2 Evaluation of the Initial Notification of the Loss of Annunciator Event
After the overhead annunciator system had been reset, the SNSS and NSS had
difficulty in deciding whether it was appropriate to declare an alert (per ECG
Section 108) or to provide a 1-hour notification to the NRC (per ECG Section
17D).
The operating crew debated 1) the appropriateness of EAL Section 17D
due to the portion of the EAL that stated "event was not ongoing at the time
of discovery" as this part of the EAL was not satisfied and 2) the
appropriateness of EAL Section 108 as the reactor was defueled and the
overhead annunciator system appeared to be operable after it was reset at
11:30 p.m.
After the operating crew had consulted with emergency preparedness
staff members and management several times by telephone, the operating crew
decided to enter EAL Section 17D at 1:12 a.m., on October 5, 1995, (about 2
hours after recognition of the loss of all annunciators).
ECG Section 17D, "Emergency Conditions Discovered After-The-Fact," requires
that "If discovery of an event/condition that had previously occurred, (event
was not ongoing at the time of discovery) which exceeded an Emergency Action
Level (EAL) but was not declared as an emergency, AND there are currently NO
adverse consequences in progress as a result of the event THEN refer to
Attachment 12 I-hour Report." The licensee has committed in its emergency
plan that this will be performed within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> (per ECG Attachment 12,
Step 3).
The inspectors assessed that this notification to the NRC was not initiated in
the time specified by the ECG.
The enforcement issue pertaining to this
failure is being addressed by the violation for the failure to promptly
declare an alert.
2.1.3 Evaluation of the Appropriateness of the Alert Declaration
The following information was found in the licensee's SERT report:
The Operations Manager arrived in the control room at 1:00 a.m.
The system engineer for the overhead annunciator system arrived in the
control room at 1:06 a~m.
The General Manager-Salem Operations and the Emergency Preparedness and
Radiological Support Manager arrived in the control room at 1:21 a.m .
- .-.
~
7
Upon arriving in the control room, the licensee managers identified above
discussed event declaration with the SNSS.
Licensee management concluded that
an alert declaration was appropriate and that the 1-ho**r report that had been
completed was not appropriate for the loss of annunciator event. At this
time, the SNSS was confident that the overhead annunciator system was
operable, but the system had not been formally declared as operable as the
system engineer had just started his diagnostic testing of the overhead
annunciator system.
The SNSS declared an alert per ECG Section 108 at
1:38 a.m.
The system engineer completed his diagnostic tests at 5:15 a.m.,
and the alert was terminated at 5:22 a.m.
The General Manager, Salem Operations provided key input into the decision to
declare the alert and stated to an inspector that he remained concerned with
system operability. The inspectors concluded that the alert declaration was
not inappropriate at the time it was made at 1:38 a.m., on October 5, 1995,
because OHA system operability remained unestablished at the time the alert
declaration was made.
2.2
Emergency Action Level Scheme
2.2.1 Appropriateness of the Loss of Annunciator EAL
As noted in Section 2.1.1, the NUMARC/NESP-0007 guidance document establishes
mode specificity to the EALs which address a loss of annunciator event. Also,
per NUMARC/NESP-0007, loss of annunciators is not an alert unless a transient
is in progress at the same time.
At the time of the October 4-5 loss of
annunciator event, the licensee's EALs covering loss of annunciator events
were not yet revised to conform to the NUMARC/NESP-0007 document.
The EALS
were based upon previous NRC guidance contained in NUREG-0654, "Criteria for
Preparation and Evaluation of Radiological Emergency Response Plans and
Preparedness in Support of Nuclear Power Plants."
After the event, the licensee discussed changing the loss of annunciator EALs
with the NRC.
The licensee informed the NRC that its 10 CFR 50.54(q) review
of the planned changes had determined that there would be no loss of
effectiveness of the emergency plan.
The licensee was specifically reminded
by NRC representatives to discuss these EAL changes with the states prior to
.their implementation ..
On Saturday October 7, 1995, the licensee implemented the revised EALs for
loss of annunciator events.
The revision adopted the NUMARC/NESP-0007
guidance.for loss of annunciator events. The inspectors considered that
~evising the EALs was an appropriate action on the part of the licensee.
Section 2.2.2 provides an assessment of the licensee's efforts in seeking
approval from NRC and the efforts in discussing with the states of Delaware
and New Jersey prior to implementation of the revised EALs.
Additionally, the
licensee has submitted a completely revised EAL scheme based upon the
NUMARC/NESP-0007 guidance for the three stations operated by Public Service
Electric and Gas to the NRC for review and approval in accordance with NRC
requirements .
_ ......
~
8
2.2.2 Discussion of the EAL Scheme With the States of New Jersey and Delaware
The licensee implemented the revised loss of annunciatcr EALs on
October 7, 1995.
The licensee neither discussed nor sought agreement with the
states prior to implementing the revised EALs.
The licensee was specifically
reminded by the NRC of the need to discuss the EAL changes and seek agreement
from the states prior to implementation of the revised EALs covering loss of
annunciator events.
During discussions with the New Jersey Bureau of Nuclear Engineering (BNE)
representatives, the inspectors were informed that the licensee informed
New Jersey BNE of the EAL change on October 11, 1995, at about 3:15 p.m.
The
New Jersey BNE representatives conveyed that they were not satisfied with
being informed of the revision to the EAL scheme after-the-fact.
The inspectors also contacted the State of Delaware Emergency Management
Agency (EMA), which had not been informed of the subject EAL revision until
October 20, 1995.
Its representatives conveyed that they had no additional
concerns over the late notification.
During an interview with the licensee's Emergency Preparedness and
Radiological Support Manager, the inspectors were informed that the licensee
had never before discussed or sought agreement with the states when revising
portions of the EAL scheme .
10 CFR 50.54(q) requires, in part, that "A licensee authorized to possess and
operate a nuclear power reactor shall follow and maintain in effect emergency
plans which meet the standards in 50.47(b) and the requirements in Appendix E
of this part."
10 CFR 50, Appendix E, Section IV. "Content of Emergency Plans," Part B.
"Assessment Actions" requires, in part, "The emergency action levels shall be
based on in-plant conditions and instrumentation in addition to on-site and
off-site monitoring.
These emergency action levels shall be discussed and
agreed on by the applicant and state and local governmental authorities and
approved by NRC.
They shall also be reviewed with the state and local
governmental authorities on an annual basis."
In summary, the failure to discuss the EAL revision with the states of New
Jersey and Delaware and acquire their agreement is assessed as a violation of
10 CFR 50.54(q) and 10 CFR 50 Appendix E. (VIO 50-272/95-81-02)
The Jnspectors determined that the factors listed below contributed to the
licensee's failure to discuss and seek agreement from the states prior to -
implementing the revised EALs.
Licensee administrative procedures did not provide adequate direction
for single EAL revisions.
This practice had not been questioned during licensee annual 10 CFR
50.54(t) reviews (audits) .
. *.
~ -~.
9
Prior to the exit meeting, the licensee provided its view regarding the
failure to discuss the revised loss of annunciator EALs with the states prior
to implementation.
The following points were provided by the licensee to the
inspectors:
1)
The pertinent requirement in 10 CFR 50, Appendix E, requires the
emergency action levels to be discussed and agreed on by the applicant
and state and local governmental authorities and approved by NRC.
The
licensee maintained that it interpreted the requirement as applicable to
an applicant for an original NRC license and not an existing licensee.
The inspectors noted that 10 CFR 50.54(q) requires a "licensee" to
follow the requirements in 10 CFR 50, Appendix E .. Therefore, the
licensee must implement Appendix E regardless of whether "applicant" or
"licensee" is used.
2)
Written agreement from the states had been received by the licensee for
the revision of the Hope Creek EAL scheme to implement the
NUMARC/NESP-0007 guidance. This agreement was in the form of an
internal BNE memo provided to PSE&G.
The licensee viewed this agreement
as providing tacit agreement of the revision made to the Salem EALs
covering loss of annunciator events as both changes simply implemented
the NUMARC guidance for loss of annunciator events.
The inspectors
assessment of this point is as follows .
The inspectors found that the Salem specific EAL changes had
neither been discussed nor agreed to prior to implementation.
-
As noted in Section 2.2.1, the licensee was specifically reminded
by NRC personnel to discuss the EAL revisions with the states
prior to implementation.
As no advance notice had been provided by the licensee, the states
had no opportunity to take internal action on the revised EALs
prior to their implementation.
NOTE:
The inspectors consider
another NRC requirement germane to this issue.
requires that "A standard emergency classification and action
level scheme, the bases of which include facility system and
effluent parameters, is in use by the nuclear facility licensee,
and State and local plans call for reliance on information
provided by facility licensees for determinations of minimum
initial off-site response measures."
In essence, states and local
governments have a vested interest in EALs developed by NRC
licensees.
The inspectors concluded that the licensee had not provided a sufficient basis
to support their contention that a violation of NRC requirements had not
occurred .
/
10
2.3
Emergency Response Organization (ERO) Augmentation
2.3.1 Emergency Operations Facility (EOF)
The activation of the EOF was not required at the alert declaration (only
required when a Site Area Emergency is declared). Certain positions in the
EOF were manned but the facility was not formally activated. The inspectors
considered EOF manning at the alert level to be within the bounds of
discretion provided by the licensee's emergency plan.
No issues concerning
the EOF were noted by the. inspectors.
2.3.2 Operations Support Center (OSC)
The OSC responders were composed of individuals who were working backshift
hours at the time of the event.
The OSC was activated in a timely manner,
after the alert was declared, at 2:13 a.m.
2.3.3 Technical Support Center (TSC)
The licensee's Emergency Plan states "It is estimated that a TSC facility can
be fully activated within about 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> following initial notification of
personnel assigned to the TSC.
This estimate is only a target value and may
vary based on initial notifications, travel, and other conditions." The
inspector's discussions with licensee emergency preparedness staff
representatives indicated that experience had demonstrated that the ERO pager
system could be activated within 15 minutes of an event classification. There
were no abnormal conditions at the time that could have understandably delayed
activation.
The alert was declared at 1:38 a.m.
The TSC was activated at 3:30 a.m.
The
TSC was manned to a minimum staffing level at 4:00 a.m.
As such, the
inspectors concluded that the TSC activation was not timely.
The licensee's Emergency Plan, Section 3, "Organization," Part 10.0, "Staffing
Commitments" provides a commitment for minimum staffing in accordance with
Supplement 1 of NUREG-0737, Table 2.
Tables 3.1 and 3.2 of the Emergency
Plan, Section 3, details licensee staffing by position. The licensee has
committed to maintain the ability to.augment its ERO with at least one
electrical engineer and one mechanical engineer within about 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />.
It took about 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> from the alert declaration to fill the TSC mechanical
and electrical engineering positions with fully qualified ERO members.
Other
-positions were not filled within an hour of the alert declaration but they
were not key ERO positions.
In summary, the licensee failed to meet its emergency plan because a qualified
electrical engineer and a qualified mechanical engineer were not available in
the TSC within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> of the alert declaration. This is considered to be a
violation of 10 CFR 50.54(q). (VIO 50-272/95-81-03)
... .. ii *
. ~,.;*.
-
..
11
2.3.4 Licensee Emergency Response Organization Notification Process
A significant number of ERO members assigned emergency notification devices
(pagers) did not respond when the pagers were activated (call-out). The
reasons for this lack of response were varied. This matter was also
identified by the licensee's SERT.
Some 20 to 30 pager holders improperly
called the station (including the control room) to find out if they really had
to respond.
One of the reasons for the lack of response was that it was
common knowledge that an emergency preparedness drill was scheduled for that
day (October 5, 1995).
The alert was declared at 1:38 a.m.
The Information Technologies Operations
Center (ITOC:
a center staffed around-the-clock responsible for pager system
-activation) received direction from the Salem Control Room at 1:45 a.m., to
initiate pager call-out.
ITOC activated the pager system at 1:54 a.m.
"A"
group pager activation started at 1:55 a.m., and "C" group pager activation
was completed at 2:07 a.m. (29 minutes after the alert declaration). The
inspectors assessed that the time to complete the pager activation was lengthy
considering that the TSC was required to be activated within about one hour of
the alert declaration.
A duty call list (a list of rotating primary responders responsible for
maintaining themselves fit-for-duty and within an appropriate travel distance
to the station) is not maintained by the licensee. The licensee emergency
preparedness staff conducted periodic pager tests with the intent to assess
system operability and to monitor its ability to staff the emergency response
facilities in a timely manner.
However, the inspectors noted that the pager
test results had not been annotated with the time it took each pager holder to
travel to the station.
There was no procedural requirement for a pager holder to respond to a call-
out.
Licensee management provided its expectations on this matter in the form
of an instruction that had been disseminated on an approximate annual basis.
The inspectors concluded that the licensee ERO call-out process was weak
because key ERO members failed to respond, ERO members called the s~~tion to
find out if they really had to respond, the time to complete the pager
activation was lengthy, a duty call list was not maintained, pager system
tests were not annotated with the time it took each pager holder to travel to
the station, and there was no procedural requirement for a pager holder to
respond to a call-out.
2.3.5 Emergency Response Organization (ERO) Augmentation Sunvnary
The OSC activation was appropriate and timely.
The TSC activation was not
timely and some key positions were filled late. Licensee management stated at
the exit meeting that the emergency preparedness issues discussed in this part
of the report would be resolved prior to restart .
-*
12
2.4
Comnunications/Reporting/Notifications
2.4.1 Notification and Reporting of Events
At about 12:20 a.m., the Plant Manager notified the NRC Senior Resident
Inspector that Salem Unit 1 was preparing to declare an alert due to the loss
of annunciators. Another call was made to the NRC Senior Resident Inspector
at 1:07 a.m., that a 1-hour report would be made to the NRC Operations Center.
The licensee's 1-hour report (see Section 2.1.2) was initiated (late) at
1:12 a.m., with the NRC Operations Center.
The alert was declared formally at 1:38 a.m.
The Initial Contact Message Form
was approved by the Emergency Coordinator at 1:41 a.m.
The licensee's
communications log indicated that the Delaware State Police were notified at
1:47 a.m., the New Jersey Office of Emergency Management (OEM) at 1:51 a.m.,
the Lower Alloways Creek Township at 2:03 a.m. (a courtesy call to be provided
within 30 minutes), and the NRC Operations Center at 2:14 a.m.
The inspectors
confirmed that the New Jersey OEM had been informed as required.
In summary,
once the event was declared as an alert, notifications to off-site response
organizations were initiated in the times specified by the emergency plan.
2.4.2 Comnunications with Off-Site Response Organizations
The inspectors found that PSE&G had informed New Jersey BNE that a "five
minute alert" (the event would be declared and terminated after 5 minutes)
would be declared.
The inspectors noted that this terminology ("five minute
alert") was not described in the licensee's emergency plan.
The inspectors
concluded that such terminology should be addressed in the emergency plan and
agreed upon by the state and local governmental authorities so that confusion
will be minimized and response from NRC and/or off-site response organizations
to an event will more likely be appropriate to the situation.
The inspectors reviewed whether PSE&G had difficulty establishing
communication with New Jersey BNE and whether PSE&G had complied with its
emergency plan in regards to communications.
1)
The inspectors verified that the New Jersey BNE received all station
status checklists generated during the emergency.
Those checklists were
sparse on details concerning the OHA system, but provided the essential
parameters concerning plant status, which included that the unit was
defueled with the spent fuel pool temperature steady at 83°F.
2)
The inspectors verified that the licensee had successfully transmitted
Emergency Response Data System (ERDS) data to the NRC .
3)
13
The inspectors reviewed whether the licensee had met its obligation in
regard to staffing the NETS line (a dedicated and controlled telephone
exchange).
A New Jersey BNE representative reported to ~he inspectors
that they had initiated two calls over the NETS line tha~ were not
answered by the licensee. At the time of the inspection, New Jersey BNE
representatives were not able to provide the time they made these calls.
Further discussion of this matter with the licensee's Emergency
Preparedness and Radiological Support Manager indicated that one of the
calls may have been to the NETS line in the TSC NRC office, which was
not occupied at the time.
The inspectors verified that the licensee's communications-related ERO
positions were filled within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> of the alert declaration. A review of
licensee records revealed that the TSC communicators arrived at 2:01 and
2:37 a.m., and that by 2:35 a.m., three communicators had arrived at the EOF.
The Emergency Preparedness liaison position was filled at 2:51 a.m.
management did not activate the TSC until 3:30 a.m.
More timely TSC
activation may have provided better management oversight and clearer
communications with outside response organizations.
The inspectors concluded that PSE&G use of the phrase "five minute alert"
hindered the understanding of the New Jersey BNE regarding the event. While
the TSC communicators arrived quickly, untimely activation of the TSC may have
impacted the quality of information between the licensee and the New Jersey
BNE because TSC management was not present within the facility in a timely
manner.
More timely TSC activation may have provided better management
oversight and clearer communications with outside response organizations.
The inspectors contacted representatives from the Salem County Office of
Emergency Services, Cumberland County Office of Emergency Management, New
Jersey Office of Emergency Management, and the Delaware Emergency Management
Agency.
In each case, these representatives conveyed that they found the
communications provided by the licensee as commendable.
The licensee's
Pmergency preparedness staff had provided these courtesy calls in addition to
the required communications.
3.0
SIGNIFICANT EVENT RESPONSE TEAM (SERT) PERFORMANCE
3.1
Emergency Preparedness Review
In the emergency preparedness area, the SERT identified the key issues
relating to the alert declaration, TSC activation, and staffing. The
inspectors also agreed with the SERT's contributing factors to these key
issues.
The inspectors identified a key difference with the SERT that pertained to the
issue of the failure to discuss and seek approval from the State of New Jersey
prior to implementing changes to the Salem EAL scheme, which is discussed in
Section 2.2.2 of this report.
The SERT did not identify this matter as an
issue because their interpretation of the pertinent NRC requirements was
fundamentally different .
.. _,
14
r
3.2
Human Performance and Procedural Review
The SER1 ::sus was primarily on emergency response issues, but the SERT
investigated human performance and procedural issues in other areas.
The
inspectors observed instances when SERT investigations could have been more
thorough.
For example, one particular procedural issue identified by the SERT
was that the abnormal OHA procedure for Unit 1 was different than the
procedure for Unit 2.
Upon finding this discrepancy, the SERT identified this
as a broken barrier, but looked no further to determine its cause.
The
inspectors found evidence of a procedural change request associated with both
units. The change request was implemented on Unit 2, but not on Unit 1. This
evidence suggested a revision control problem that could have been explored in
more detail.
The SERT performed a detailed analysis of the corrective actions from the
Salem Unit 2 loss of OHA event in 1992.
The SERT aggressively identified
missed opportunities to prevent the recent event by failure to implement
certain recommended corrective actions.
The SERT performed a broad-based comparison of this event with the Hope Creek
shutdown cooling event and other industry-related annunciator events.
The
inspectors noted that this comparison showed good use of operational
experience to extract technical and organizational insights.
3.3
Integration and Usability of Operational Procedures
The SERT found that the operational procedures for the OHA were inadequate. A
particular weakness observed by the inspectors was the apparent collective
inefficacy of the engineering, procedure writing, and training organizations
to support the plant operations staff with an integrated, consistent, fully
usable tool kit for the determination of OHA availability. The individual
procedures for the OHA by the organizations were generally very good, but the
collective set did not fully meet the usability needs of the operators.
The operators used two procedures to diagnose and correct the loss of OHA
during the event and the Event Classification Guide to determine emergency
response actions. The first procedure, Sl.OP-SO.ANN-001, Rev. 7, was the
Normal Operating Overhead Annunciators Procedure. The**operators primarily
used only two sections from this procedure during the event: (1) Testing the
Operability of the OHA System and (2) Resetting the OHA System.
The second
procedure, Sl.OP-AB.ANN-0001, Rev. 4, was the Abnormal Procedure for Overhead
Annunciator Operation.
Examples of the lack of integration and operator usability.reflected in the
procedures are summarized as follows:
There was inadequate consideration of the 15 minute emergency
classification requirement in the ECG in the development of the Normal
and Abnormal Operating Procedures.
The steps required by these
procedures to determine system operability and restore operability took
longer than 15 minutes for the crew to perform .
,,
15
Engineering and training relied heavily on symptoms of failure from the
1992 event.
Operators were not specifically trained on other
indications of systeLl failure.
One example of th,s was that the
indications of an incomplete failover from SER A to SER B were not
specifically covered.
In this event, the operators did not recognize
that the failover did not occur. Another example is that the operators
expected the clock on the CRT to stop if the system was inoperable.
In
this event the clock continued to update.
Operators were observing
inconsistent indications of system operability.
A trouble annunciator was installed after 1992 event to alert operators
to OHA failure.
When this panel annunciator is actuated, overhead
annunciator window A-9 is illuminated and an audible alarm sounds.
This
window was not actuated during the event.
Engineering intent was not correctly translated into operating
procedures.
Upon loss of OHA, engineering intended the operators to
switch manually from SER-A to SER-B.
However, the way the procedures
were written, the operators were trained to reset the system rather than
to manually switch the SERs.
Procedural tests were too long and cumbersome for operators to quickly
determine system status, so operators sometimes relied on the "Lamp
Test" to determine system status rather than use procedural methods .
The abnormal procedure revision process was not adequately controlled .
Salem Unit 1 Abnormal Procedure directed operators to reset the OHA
system upon any indication of loss of OHA.
The operators were not clear
on when to use the abnormal procedure.
Salem Unit 2 Abnormal Procedure
directed operators to test first for operability and then reset only if
required, which was different than Unit 1.
During the event, SER-A failed but due to the nature of the failure, the
OHA system did not failover to SER-B as designed.
When the system does
failover, there is indication to the operating crew by the extinguishing
of all previously lit window boxes, then a reflash followed by a
restoration of all alarmed windows (repainting). Although this
indication was known by the engineering staff and included in the
discussion section of the normal operating procedure, operators were not
specifically trained to look for this indication. Since the operators
were not trained to look for a failover, they did not realize that the
system had not failed over to SER-B.
Had the operators been trained
more aggressively on system operations and indications, they may have
realized that the system failed to successfully failover and then taken
action to manually switch control from SER-A to SER-B as engineering
intended for this particular situation .
~
16
'
3.4
Overall SERT
The inspectors considered the licensee efforts in highlighting emergency
preparedness problems to be commendable, even though the inspectors did not
reach the same conclusions in all cases.
The inspectors agreed, in general, with the conclusions and recommendations
for human performance and procedural issues.
In some cases, recommendations
were not clearly stated. Problem areas were identified, but specific
corrective actions were not described. A particular weakness observed by the
inspectors was the apparent collective inefficacy of the engineering,
procedure writing, ~nd training organizations to support the plant operations
staff with an integrated, consistent, fully usable tool kit for the
determination of OHA availability.
4.0
TECHNICAL ROOT CAUSE PERFORMANCE
4.1
Causal Factors Analysis
The licensee postulated failure modes that were developed into scenarios,
which were then examined for the most likely failure mechanism.
Causal
factors analyses were performed for four human activity scenarios, thirteen
hardware failure scenarios, and fifteen software failure scenarios.
The human activity scenarios covered such items as operator error, welding
activity in the area, and inadvertent grounding of the field contact power.
The licensee concluded that human activity was not a likely initiator of the
event.
The hardware failure scenarios analyzed areas such as induced electrical
noise, power supply problems, failover switching, and data line switching.
The licensee concluded that there were no likely hardware scenarios analyzed
that would cause the system failure indications. However, induced noise and
some type of input power spike affecting the SER circuits were not ruled out
as likely initiators that could cause the hardware to induce a software fault.
The software failure scenarios covered software faults in all major programs.
The three most likely scenarios centered on SER A in the scanner
acknowledgement, memory, task error, and task failure areas.
The event-expand
task,, which processes the data from the scanners was singled out as the most
likely software fault candidate that matched the lack of failover and other
indications. The vendor analysis indicated that the event-expand task
probably exited or was aborted due to an undetermined cause.
The licensee
conducted independent tests at the vendor facility that confirmed the results
of the event-expand task analyses.
The licensee also conducted a design
review of the vendor software by examining the system architecture and the
software code with the software developers. This review uncovered a design
flaw that concerned the placement of the watchdog timer refresh commands in
the task structure, which could prevent failover under certain conditions .
l
J
.. ..
(~
,.
17
f e
The inspectors' evaluation was that the causal analysis failure scenarios were
accurate, detailed, reasonable and thorough.
Detailed statements of
supporting evidence and refuting evidence supported each conclusion.
4.2
Root Cause
The root cause subteam determined that the failure was most 1ike1 y caused by
the undetected premature exit of a single software task in SER A, which halted
the processing of alarms to the OHA window boxes.
The exact initiating event
was not determined, but some combination of possible degradation of the SER
power supplies and power line instability was implicated.
The determination
of the most likely root cause was technically sound and was the best fit with
the actual indications of failure.
According to 10 CFR 50.47(b)(8), it is a requirement that adequate emergency
facilities and equipment to support the response be provided and maintained.
The OHA system is part of the emergency equipment to support the response and
can alert operators to abnormal conditions, but operators must be able to
determine whether the OHA system is available to implement the ECG.
The root
cause for this event was an OHA software task fault that halted the processing
of alarms with no indication, alarm, or annunciation to the plant operator.
This is the second occurrence of this type of "silent" malfunction without
failover to the backup, which made the equipment unavailable to perform the
OHA function, and did not provide the operators with sufficient information to
determine that unavailability. Therefore, the licensee provided inadequate
emergency equipment to support emergency response, which is a violation of 10
CFR 50.47(b)(8) (VIO 50-272/95-81-04, 50-311/95-81-04).
4.3
Operability Determination and Compensatory Periodic Tests
The inspectors reviewed the document, "Operability of Overhead Annunciator Due
To a Silent Fault," Number SES95-441, dated November 7, 1995.
Salem Unit 1
was defueled and Unit 2 was in mode 5, with plans to transition to mode 6 and
defueling at a later date. The document included design basis requirements,
analysis/assessment, specific operability requirements, and
conclusions/actions.
The licensee conclusion was that the OHA systems were
operable, but in a degraded mode.
The follow-up assessment of operability,
after corrective action, was to be tracked by the licensee as a Unit startup
issue.
Compensatory manual tests were performed, in addition to the 8-hour and weekly
tests, by the operators on a periodic basis. The licensee stated that
compensatory tests should detect the known software failure mechanisms that
could result in non-operability. A test performed every 30 minutes used an
alarmed cabinet door as an input alarm condition.
The licensee stated that
this test should check the ability of OHA to process data through a scanner,
through the SER A, and display the alarm/return to normal state on the
appropriate window, CRT, and printer. Another test performed every 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />,
called the lamp test, should test all the window lamps and inject 11 known
alarms into three scanners.
The lamp test should verify the same conditions
as the alarmed door test with the additional test of two scanners.
..._ ........ -
...
'
18
The inspectors noted that the operability determination covered the possible
loss of core cooling in Unit 2, and the possible radiological alarms in the
spent fuel pool area of Unit 1.
The finding of the root cause subteam that
there was a known class of software faults that could go undetected was also
factored into the analysis.
The technical analysis of the ability of the
compensatory tests to detect the known class of software faults was very good.
The inspectors concluded that the operability determination, with the
compensatory tests specified, was sufficient for use in the respective plant
modes.
4.4
Comparison With the 1992 Salem Unit 2 OHA Event
The licensee performed a detailed comparison of the 1992 Salem Unit 2 OHA
event with this 1995 Salem Unit 1 OHA event.
The 1992 event was centered
around operator errors with manual keyboard strokes and interface switch
settings, which in conjunction with a software download task, suspended
certain other SER A tasks. The vendor software design was such that with the
unexpected switch positions and keyboard errors, the download task was waiting
for data that never came.
One of the tasks that was not suspended was the
update of the watchdog timer, which in turn did not allow the transfer of
alarm processing to SER B.
The result was that the OHA did not process alarms
for 90 minutes, until discovered by an alert operator.
The OHA system did not
indicate that the alarm processing had stopped.
For the 1995 event, the root cause subteam ruled out operator errors
concerning keyboard strokes and interface switch positions as an initiator.
After the 1992 event, the keyboard was placed in a locked compartment and had
administrative controls for access to the key.
The keys were not checked out
before or during the 1995 event. Also after the 1992 event, the manual
interface switch function was performed by an electrically interlocked switch
that aligned the data paths correctly.
In the 1992 event, the operators did not follow procedures, which allowed some
troubleshooting data to be lost.
In the 1995 event, the operators followed
procedures, which allowed them to reset the SERs; the reset caused some data
to be lost. However, the system capability for storing historical data was
used in the diagnosis of the software fault.
The print-outs provided a record
of missed data items that could be interpreted for correct or incorrect
operation. Special data printers added after the 1992 event to collect a
reduced set of error codes and SER status were not effective in providing
diagnostic data for the class of software faults suspected.
The licensee concluded that a contributing factor to the 1995 event was that
the apparent scope of corrective actions after the 1992 event was too narrow.
Those--actions were aimed at preventing only the set of errors that were
present and detectable. The emphasis was not on finding the adequacy of how
the system detected possible faulted conditions and indicated or took action
in response to those faults to prevent failure of alarm processing.
A
deterrent effect to widening the scope was that the vendor, as determined by
the licensee, had no design specifications for the software and relied on the
collective memory of the software designers through commented code for the
19
architectural details. The licensee determined that the vendor software
design process, procedures, and documentation were less than adequate, which
also hampered the effectiveness of corrective actions from the 1992 event.
The main similarity between the 1992 and 1995 events was in the inability of
the design to detect a certain class of software faults that would cause a
failure of the OHA to process valid alarms.
In the 1992 event, the operator
error sequence initiated the suspension of a task that was not detected and
failover did not occur.
In the 1995 event, a suspected power supply problem
was a strongly implicated initiator for the suspension of a different task
that was not detected and for which no failover occurred.
In both cases, the
particular software fault was not detected or indicated.
4.5
Generic Implications and Notifications
Every vendor annunciator system of the type used in the Salem units may be
vulnerable to the type of silent failure experienced.
Those systems that
incorporate the same system architecture as Salem plants, the redundant
failover pair, may be the most vulnerable to loss of annunciator function.
For those systems that have a redundant train architecture, each train may be
vulnerable to the same type of silent failure, but the vulnerability to
complete loss of annunciator function is less than the redundant failover pair
architecture.
The NRC residents at licensee sites with the basic vendor equipment,
regardless of system architecture, were notified by Region I.
The sites were
Kewaunee, Sequoyah, and Pilgrim.
The engineering staff at PSE&G notified the
industry by issuing Operating Experience Information Notice Number 7575 on
November 15, 1995.
The vendor, Hathaway Process Instrumentation, issued
Quality Alert, Issue Number 116, on November 6, 1995, to all customers who
purchased similar equipment.
5.0
MANAGEMENT OVERSIGHT
The management provided tight oversight of the decision to declare the alert.
The management chartered the SERT team.
The SERT team leader was a ma11ager,
and two managers were overseeing the root cause team.
The SERT process and
report was reviewed by management.
The management provided outside failure
analysis consultants to both the root cause and SERT subteams.
The inspectors
inferred from these actions that management was actively involved in the alert
declaration and overall direction of the failure analysis process.
6.0
OVERALL CONCLUSIONS
- Although the conditions for an alert declaration had been exceeded and
identified by the operating crew, the operating crew decided not to
declare an alert. This was assessed to be a violation of
10 CFR 50.54(q) (VIO 50-272/95-81-01).
20
The licensee's failure to discuss and seek agreement with the States
. prior to the implementation of the revised EALs covering loss of
annunciator events was assessed to be a violation of IO CFR 50.54(q) and
IO CFR 50 Appendix E (VIO 50-272/95-81-02).
TSC activation was not timely.
The licensee did not meet the emergency
plan staffing requirements for the TSC.
This was assessed to be a
violation of IO CFR 50.54(q) (VIO 50-272/95-81-03).
There was no indication to operators of major system problems that would
prevent alarm processing. This type of failure also occurred in the
I992 Salem Unit 2 OHA event. This is the second occurrence of this type
of "silent" malfunction without failover to the backup, which made the
equipment unavailable to perform the OHA function, and did not provide
the operators with sufficient information to determine that
unavailability.
This a violation of IO CFR 50.47(b)(8) because the
licensee provided inadequate emergency equipment to support emergency
response in that the declaration of an alert is based on the
availability of the OHA system (VIO 50-272/95-81-04, 50-311/95-01-04).
The event was initially dispositioned by the operating crew as a I-hour
report. This I-hour report was not initiated in a timely manner.
OSC activation was timely.
EOF manning was a conservative act on the
part of the licensee and was within the level of discretion provided by
the NRC-approved emergency plan.
The licensee's action to change the loss of annunciator EALs to be mode
specific was appropriate based on NRC's generic acceptance of the
NUMARC/NESP-0007 EAL guidance document.
The licensee's ERO ca 11-out process was weak.
The 1-hour report and the alert were not made ir; a timely manner;
however, the subsequent reporting requirements were carried out in a
timely manner.
The informal communications hindered understanding of the event for the
State of New Jersey Bureau of Nuclear Engineering (BNE).
Otherwise,
representatives from other offsite response organizations found the
communications provided by the licensee to be commendable.
The determination of the most likely root cause was technically sound
and was the best fit with the actual indications of failure.
The operability determination and the compensatory tests were sufficient
for use in the respective plant modes.
..
\\_.
..
7.0
21
There was an overall weakness observed in the effectiveness of the
engineering, operations, and training organizations to support the plant
operators with a unified set of OHA knowledge, skills, a~~ abilities to
recognize failure indications, to determine operability, and to take
proper corrective action.
Operator errors in keystrokes and interface switch settings were not a
factor in this event due to effective corrective actions for these areas
taken after the 1992 Salem Unit 2 OHA event.
Management was actively involved in the alert declaration and overall
direction of the failure analysis process.
EXIT MEETING AND TELEPHONE CALLS
An exit meeting was held on November 13, 1995, with the PSE&G personnel as
noted in Attachment 2 to summarize the scope and findings of their inspection
activities. The licensee acknowledged the inspection findings and also had
some comments beforehand as discussed in Section 2.2.2 of this report.
The
inspectors neither received or reviewed any proprietary material during the
inspection.
A telephone conference was held on November 17, 1995, with
Messrs. Munzenmaier, Villar, and Banner to resolve the comments from the exit
meeting.
The Regional State Liaison Officer called the New Jersey BNE on
January 30, 1996, to confirm the accuracy of the references to New Jersey BNE
contained in the report. During that call, the New Jersey BNE representative
stated that the licensee's station status checklists (see Section 2.4.2) did
not contain sufficient information to serve as a basis for de-escalation of
State resources allocated to respond to the loss of OHA event.
Attachments:
1.
Special Reactive Inspection Charter for the Salem 1 Alert on
October 5, 1995
2.
Exit Meeting Attendees
-.
-~
ATTACHMENT 1
CHARTER
S?E2I~l REACTIVE INSPECTION OF THE SALEM 1 ALERT ON OCTOBER 5, 1995
I.
Overhead Annunciator Performance
Independently evaluate licensee's root cause(s) of loss of
annunciation.
Assess any previous opportunities to detect and correct problem.
Evaluate adequacy of licensee's current corrective actions.
Evaluate any commonality with the previous Unit 2 loss of
annunciator event.
Evaluate initial actions with overhead annunciator system to
preserve data for future troubleshooting.
II.
Evaluate operator actions and decisions relating to implementing
the Salem Emergency Classification Guide (ECG).
Assess adequacy of emergency plan to address the situation.
Evaluate implementation of emergency plan, including TSC staffing.
. 4 * *
ATTACHMENT 2
EXIT MEETING ATTENDEES
public Service Electric and Gas
C. Bakken
M. Bursztein
N. Coni cell a
T. DiGuiseppi
C. Fricker
C. Munzenmaier
P. Noeller
L. Raj kows k i
M. Renchek
L. Storz
E. Villar
C. Waite
C. Warren
Atlantic Enerqv
J. Lazzara
Delmarva Power
P. Duca
Manager, Salem Operations
Manager, Nuclear Electrical Engineering
Manager, Salem Restart
Manager, Radiation Safety
Supervisor, Salem Plant Assessment
General Manager, Nuclear Operations Services
Licensing
Manager, Salem Main~enance, Controls
Manager, Salem System Engineering
Senior Vice President, Nuclear Operations
Licensing Engineer
Supervisor, Digital Systems Group
General Manager, Salem Operations
Site Representative
Site Representative
Philadelphia Electric Company
R. Kankus
Senior Strategic Planning Specialist, Joint Owners
Alliance
Delaware Emergency Management Agency
K. Kil te
Radiological Emergency Preparedness Specialist
New Jersey Department of Environmental Protection
P. Gardner
T. Kolesnik
J. Li pot i
P. Mulligan
Research Scientist
Nuclear Engineer
Assistant Director
Radiation Physicist
U. S. Nuclear Regulatory Commission
C. Marschall
L. Nicholson
W. Ruland
Senior Resident Inspector
Branch Chief, Division of Reactor Projects
Branch Chief, Division of Reactor Safety
--~.