ML20203J514
| ML20203J514 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 02/26/1998 |
| From: | Todaro R PACIFIC GAS & ELECTRIC CO. |
| To: | |
| Shared Package | |
| ML20203J517 | List: |
| References | |
| LER-98-S02, LER-98-S2, NUDOCS 9803040169 | |
| Download: ML20203J514 (9) | |
Text
'
I LICENSEE EVENT REPORT (LER) f ACILITV NAME (t)
DocktiNuvsFR p) l F AGE < ni Diablo Canyon Unit 1 0
5 0
0 0
2 7
5 1
oF g
IlfLE (4)
Following Security Computer System Failure, Full Compensatory Measures Not f
implemented Within 10 Minutes Due to Personnel Error DAT('
Jh"R (m "M"A MON DAY YR YR SEiJUENTIAL NUMBER R S MON' DAY YR F ACILITY NAME DOCKET NUMBER 01 27 98 98 S 0!2 0
0 2
26 98 Diablo Canyon Unit 2 0
5 0
0 0
3 2
3 OPERATING THIS REPORT IS SUBMITTED PURSUANI TO THE REQUIREMENIS OF 10 CFR (11) 1 X
10 CFR 73.71(d) tivT, ",
OTHER 0
0
( S"' * '" *" S '"^ ' S " * ^" ' " ' ' * ' ' " " ' " " ' " ^ '
LICENSE E CONTACT FOR THIS LE H (12) 1E t E PHONE NUMBE N Ronald G. Todaro - Director, Security Services 805 545-4309 COMPI E TE ONE LINE FOR E ACH LOMPQ6aE NT F AltuRE DE SCRIBE D IN fHIS REPORT (13p CAUSE SYSTEM COMPONENT MANutACTURER RE PORIA LE CAUR SYSTEM COMPONENT MANUF ACTURER REPOR T A8tE X
llD lDlC.!C 2l4l2lF N
l l
l l
l l
l l
l l
l l
l l
l l
l l
l l
l SUPPLEMENTAL Rf PORT EXPECTED (14) l EXPECTED MON DAY M
[ ] YEs (If yes. complete EXPECTED SUBMisslON DATE)
[X] NO SUBMISSION DATE (15)
~ ABSTRACT (top On January 27,1998, at 1434 PST, with Units 1 and 2 in Mode 1 (Power Opuation),
at 100 percent power, the security alarm annunciation was lost because the security computer system failed. Security officers were dispatched and directed to implement compensatory measures. At 1455 PST, alarm and cardreader functions were i
restoied to certicc. At 1516 PST, a 1-hour, non-emergency report was made in accordance with 10 CFR 73.71(b)(1). At 1045 PST, it was discovered that one officer had not properly implemented compensatory measures. By 1720 PST, security and operations personnel had completed inspection of all areas affected by the failed compensatory action. On January 28,1998, at 0004 PST, computer system recovery testing was completed.
Or. February 3,1998, at 0928 PST, the 1-hour non-ernergency report was updated to report that one officer had not properly implemented compensatory measures.
The computer systein failure was due to a failed disk drive. The officer's failure to properly implement compensatory measures was due to personnel eiror (cognitive).
Corrective actions include replacement of disk drives, review of security computer maintenance plans, auditing of response cards, and additional training for response personnel.
9903040169 980226 PDR ADOCK 05000275 S
PDR 1
J
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION uaum,wt. i, oocu i,wio a, gay +.
eu.t o,
,-o,.
,-o,.
Diablo Canyon Unit i 0
5 0
0 0
2 7
5 98 S
0 2
0 0
2 OF g
itzt 1.
Plant Conditions Unit 1 and Unit 2 were in Mode 1 (Power Operation) at 100 percent power.
II.
Description of Problem A.
Summary On January 27,1998, at 1434 PST, with Units 1 and 2 in Mode 1 at 100 p.srcent power, the security alarm annunciation was lost because the security computer (CPU) system failed. Security officers were dispatched and directed to implement compensatory measures. At 1455 PST, alarm and cardreader functions were restored to service. At 1516 PST, a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />, non-emergency report was made to the NRC in accordance with 10 t'FR 73.71(b)(1). At 1645 PST, it was discovered that one officer had not properly implemented compensatory measures. By 1720 PST, security and operations personnel had completed inspection of all areas affected by the failed compensatory action. On January 28,1998, at 0004 PST, computer system recovery testing was completed.
On February 3,1998, at 0928 PST, the 1-hour non-emergency report was updated to report that one officer had not properly implemented compensatory measures.
B.
Background
The Diablo Canyon Power Plant (DCPP) Physical Security Plan, Section 3.1.4.5, identifies intrusion alarm systems and states, in part," Failure of any part of the system results in implementation of appropriate comrr hsatory measures."
DCPPt a Procedure, SP 106, Section 6.4.8.a, " Loss of Alann Annunciate ivent 6F)," states the following:
This ever zust bo reported within one hour of occurrence if:
The c ginal alarm annunciation capability is not restored within 10 minutes, or security personnel with appropriate communications are not in o
place within 10 minutes to compensate for the loss.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION we at o, muun a a,
,;a.o ex.t m g...
Diablo Canyon Unit i 0
5 0
0 0
2 7
5 98 S
0 2 -l0 0
3 0"
9 18X7 The security system remote multiplexer units (RMUs) continue to receive data from security system field devices even though the computer system is off line. Upon retuni to service, the data held by the RMUs is reported to the computer and appropriate actions are taken by the central alarm station operator.
On January 10,1998, the security department implemented a new response plan for security system outages. Training and drills were conducted in December of 1997.
C.
Event Description On January 10,1998, the security and technical maintenance (TM) organizations conducted annual security system maintenance activities. Two history disk drives were replaced, in accordance
[
with a commitment made after a previous computer system failure in May of 1997 (Fief. LER 197-S02). These drives were reformatted and tested in the security lab system just prior to the installation. New disk drives are no longer available from the manufacturer.
On January 13,1998, at 1000 PST, a secondary disk failure occurred. The security computer system continued to operate.
One of the history disk drives (DZ1) installed on January 10,1998, was believed to be the failure and was replaced on January 14, 1998 On January 15,1998, at 1027 PST, a secondary disk failure occurred causing a loss of the security computer system. The history disk drive (DZ1) installed on January 14,1998, was believed to be the most likely cause of the system failure was taken off-line. Alarm and cardreader functions were restored at 1306 PST. A 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> non emergency report was made for this event.
The report was retracted on February 5,1998, after it was venfied that appropriate compensatory measures were initiated within 10 minutes.
On January 15,1998, at 1002 PST, the main security computer failed, causing a loss of alarm and cardreader functions.
Compensatory measures were still in effect from the 1027 PST
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION
.um= o, u,mm a a, gang 7 em o, Nt M9f R NtWDF R Diablo Canyon Unit 1 0
5 0
0 0
2 7
5 98 0
0 4
OF l g
S 0
2 nn event. The backup security computer took over as designed. No cause could be identified for this loss. Testing was completed at 2033 PST.
On January 16,1998, the suspect history disk drive (DZ1) was replaced with a newly formatted and tested disk drive.
On January 24,1998, at 2152 PST, the security computer system experienced an unexpected system transfer. The main and backup i
computers transferred as designed. The cause for this transfer could not be identified.
On January 27,1998, at 1434 PST, a secondary disk failure occurred causing a loss of the security computer system. The central alarm station operator immediately initiated compensatory measures for the loss by dispatching security officers to strategic areas of the plant. The video capture system continued to report alarms for the main and intake protected area perimeter detection zones. At 1455 PST, alarm and cardreader functions were restored.
On January 27,1998, at 1516 PST, a 1-hour non emergency report was made to the NRC in accordance with 10 CFR 73.71(b)(1). The operations shift supervisor was also advised of the event.
On January 27,1998, at 1645 PST, it was discovered that one compensatory measure had not been correctly implemented. One security officer failed to perform the correct compensatory action because he used the wrong response card. The officer did not checl. the correct eleven (11) doors assigned to his area. A subsequent investigation revealed that six (6) of the doors had been checked and determined to be secure by other officers.
On January 27,1998, at 1720 PST, security and operations personnel completed an inspection of all areas associated with the failed compensatory measure. No adverse conditions were discovered. No activity for the unchecked doors was reported from the RMUs upon system recovery.
On January 27,1998, at 2011 PST, the security system computers were restored to normal operation. Both history disk drives (DZ1
LICENSEE 2 VENT REPORT (LER) TEXT CONTINUATION mw o.e v, um mun a, n+,
va a, MI AM R NtMM ot Diablo Canyon Unit 1 0
5 0
0 0
2 7
5 98 S
0 2
0 0
5 OF 9
ftKT and DZ5) were removed from the operating security system and placed in the security lab system for further analysis. Newly reformatted history disk drives were install 6d in the operating security computer system. No additional failures have been observed to date.
On January 28,1998, at 0004 PST, testing was ccmpleted verifying all zones associated with the security computer system were functioning properly. Compensatory measures were released.
On February 3,1998, at 0928 PST, the 1-hour non-emergency report was updated to report an improper compensatory response to the event.
PG&E believes the cause of the initial disk drive failure on January 13,1998, was mainte -Jnce induced. Two history disk drives (DZ1 and DZ5) were replaced on January 10,1998. The fncus of the maintenance activity from January 13,1998, through January 27,1998, was toward history disk drive DZ1. The other history disk drive (DZ5) had an unrecognized intermittent failure that was not discovered until February 19,1998, while under diagnostic conditions in the security lab system.
D.
Inoperable Structures, Components, or Systems that Contributed to the Event None.
E.
Dates and Approximate Times for Major Occurrences 1.
January 27,1998, at 1434 PST: Event date: The security computer system failed.
1.
January 27,1998, at 1516 PST: A 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />, non-emergency report was made to the NRC in accordance with 10 CFR 73.71(b)(1).
?
w,. ~
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION
.,,,,,w,
-, <., w,
gg.p
.m, Die')lo Canyon Unit i 0
5 0
0 0
2 7
5 98 S
0 2
0 0
0 OF 9
ftsf 3.
January 27,1998, at 1645 PST: Discovery date: PG&E discovered one officer did not properly implement compensatory measures.
F.
Other Systems or Secondary Functions Affected None.
G.
Method of Discovery Visible indications on the alarm station console identified the security computer system failure.
H.
Operator Actions None.
l.
Safety System Responses None.
Ill.
Cause of the Problem A.
Immediate Causes 1.
A disk drive failed on the security computer system.
2.
A utility security officer performed incorrect compensatory actions.
B.
Root Causes 1.
The initial failure on January 13,1998, caused data to be corrupted on the history disk drives. The corrupted data was unknowingly copied to each successive replacement disk drive until its discovery on January 27,1998. The problems observed on the operating security system were duplicated in the security lab.
LICENSEE EVENT REPORT (LER) TEXT CONTINUATION u,v, ~~
gg-
$d!MHf R br.Nfit R Diablo Canyon Unit 1 0
5 0
0 0
2 7
5 98 S
0 2
0 0
7 OF g
TEaf 2.
The security officer's failure to implement correct compensatory actions was due to personnel error (cognitive). Though the officer had been trained, he did not recognize that the card being used was incorrect.
C.
Contributory Causes 1.
Missed communications and inattention to detail caused TM personnel to focus attention on the wrong disk drive. TM used only visual indicators to determine equipment loss and did not refer to security system advisories.
2.
As a new response plan was in effect, the security officer was in possession of multiple door response cards, one of which should not have been in the packet.
3.
Preventive maintenance to routinely replace disk drives is believed to be another contributory cause. Whenever an operating electronic component is replaced, there is an increased risk of disk failure in the near future. Electronic l
components have a higher probability of failure when they are disturbed.
IV.
Analysis of the Event l
Security officers were immediately dispatched to strategic areas of the l
plant to perform compensatory measures. Additionally, the main and intake protected area video capture system continued to function during the event. Upon discovery that one officer had performed an incorrect response, security and operations personnelinspected all areas that had not been properly compensated No adverse conditions were found. All affected vital area doors were inspected and tested with satisfactory
- results, l
PG&E determined there was no malevolent intent involved in the computer system failure and that no adverse conditions resulted from the delay in establishing full compensatory measures.
Thus, the health and safety of the public were not affected by this event.
l
. LICENSEE EVENT REPORT (LER) TEXT CONTINUATION
. m,r-o,,
on -m a, g,,;. +
,~
Diablo Canyon Unit 1 0
5 0
0 0
2 7
5 98 S
0 2
0 0
8 OF 9
ftAf V.
Corrective Actions A.
Immediate Corrective Actions 1.
Security officers were immediately dispatched to strategic areas of the plant to perform compensatory measures.
2.
Interim instructions were provided to security officers on the correct response card to use.
3.
An audit of response cards was conducted to ensure only appropriate cards were in use. Discrepancies were corrected.
B.
Corrective Actions to Prevent Recurrence 1.
TM will review its response to security system computer failures.
2.
Security and TM will review and revise the security computer preventive maintenance activities.
3.
If necessary, the corrective actions taken to prevent recurrence described in LER 1-97 S02 will be revised (Ref, the following discussion on previous LERs).
4.
Additional training will be provided for each security officer classification emphasizing proper response to a security system outage.
VI.
Additional information A.
Failed Components Security computer disk drive Manufacturer:
Fujitsu Ltd.
Model Number:
M2351 i
. LICENSEE EVENT REPORT (LER) TEXT CONTINUATION
, om.
- -m m Diablo Canyon Unit i 0
5 0
0 0
2 7
5 98 S
0 2
0 0
9 OF g
fl af B.
Previous LERs on Similar Problems LER 1-97-S02," Security Computer System Failure Due to the Apparent Failure of the Computer Disk Drive, Without implementing Full Compensatory Measures Within 10 Minutes."
The root cause for the computer failure on May 10,1997, was reported as bearing wear on a disk drive resulting in head contact with the disk surface. Both program disk drives were replaced and the history disk drives were purged of all data. The actions taken to prevent recurrence necessitate replacement of two of the four disk drives every year to lessen the probability of a similar failure in the future. PG&E now believes that replacing the disk drives was the initiator for the current event.
A contributory cause discussed in LER 197 S02 for exceeding the 10-miriute requirement to implement full compensatory measures was an insufficient number of officers available to respond during plant outage activities. The actions taken to prevent recurrence were to review and revise the computer system failurs response plan, and conduct on shift drills, in the current event, the failure to implement full compensatory measures was personnel error by one security officer and not the result of an insufficient number of officers. Therefore, the corrective actions to prevent recurrence described in LER 1-97-002 were effective and would not have prevented the current event.
_ - _ - - _