IR 05000293/1987053
| ML20237C423 | |
| Person / Time | |
|---|---|
| Site: | Pilgrim |
| Issue date: | 12/14/1987 |
| From: | Baranowsky P, Chiramal M, Durr J, Eichenholtz H, Haverkamp D, Knox J, Ruscitto D, Wessman R NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20237C407 | List: |
| References | |
| 50-293-87-53, NUDOCS 8712210308 | |
| Download: ML20237C423 (101) | |
Text
{{#Wiki_filter:_- _ _-. _ _ . U.S. NUCLEAR REGULATORY C0fiMISSION
REGION I
Report N /87-53 Docket N /87-53 License No. DPR-35 Priority - Category C Licensee: Boston Edison Company 800 Boylston Street Boston, Massachusetts 02199 Facility Nane: Pilorim Nuclear Power Station Inspection At: Plymouth, Massachusetts l Inspection Conducted: November 16-20, 1987 l Inspectors: l J. P. Durr, Team Leader, RI date , D. R. Haverkamp, Assistant Team Leader, RI date
$ 44A4ttff)
P. W. Baranowsky, ean Member, NRR
/7/2lf7-
'date V /2-/ > / 87 M. Chirama,1, Team Member, AEOD .date
-
,
.
H. Eichenholtz, Tean Member, RI date Qada J./L.~ Knox,TeamMember,NRR nhh, dat'e CN R. H. Wessman, Team Member, NRR it /t/qw
**
date[ l Approved by: l W. V. Johnston, Director ( Acting) DRS date Inspection Sumary: See Executive Sunmary Areas Inspected: 8712210308 871214 PDR l Results: G ADOCK 05000293 PDR I
- _ _ _ _ - _ _ _ -
- _ _ - _ _ _
l
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
l ' Report N /87-53 Docket N ! License No. DPR-35 Priority - Category C Licensee: Boston Edison Company 800 Boylston Street Boston, Massachusetts 02199 Facility Name: Pilgrim Nuclear Power Station Inspection At: Plymouth, Massachusetts Inspection Conducted: November 16-20, 1987 Inspectors: ow /,2/N/f 7 (). P. Durr, Team Leader, RI ' dMe
. $b w am/
D. R. Haverkamp, A/sistant Team Leader, RI
/2/t Y O' ?
date Or'c'...'..' .vn+.. . . ,,; y P. W. Baranowsky, Team Member, NRR ' d' ate ~ Oriut=1Dicu; L;: ,y/4/g 7 M. Chiramal, Team Member, AEOD 'date h^ fx ,y 4 O//4 /0
' date'
'
H.Eichenholz,TeamMembg,RI Orl;ul i :g:ae,1:y: p J. L. Knox, Team Member, NRR ta(e MO /Jkl77 (D' G. Ruscitto, Teamd4 ember, RI ' date ' Orici=1 SIcncc Ir: jg/aff7 R. H. Wessman, Team Member, NRR 'date Inspection Summary: See Executive Summary _ _ - _ _ _ _ -
r- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - l l l , l' i I
' -
SUMMARY OF RECOMMENDATIONS . . . . . . . . . . . . . . . . . 2 3.0 EVENT DESCRIPTION . . . . . . . . . . . . . . . . . . . . . . 3 4.0 EQUIPMENT PERFORMANCE . . . . . . . . . . . . . . . . . . . 11 4.1 ELECTRICAL SYSTEMS . . . . . . . . . . . . . . . . . . . . . 11 4.1.1 345 kV ELECTRICAL OFFSITE POWER SOURCE . . . . . . . . . 11 4.1.1.1 SYSTEM DESCRIPTION . . . . . . . . . . . . . . . . . 11 4.1.1.2 OPERATIONS AND FAILURES DURING EVENT . . . . . . . . 12 4.1.1.2.1 STARTUP TRANSFORMER . . . . . . ...... . 14 4.1.1.2.2 STUCK BREAKER . . . . . . . . . . . . . . . . . 15 4.1.1.2.3 SWINGING / OSCILLATING TRANSMISSION LINES . . . . 16 4.1.1. 2 . 4 I N SU LATORS . . . . . . . . . . . . . . . . . . .. 16 4.1.1.3 REC 0VERY . .. .............. .. 17 4.1.1.3.1 WASHING OF SWITCHYARD . . . . . . . . . . . . . 18 4.1.1.3.2 BACKFEED THROUGH UNIT AUXILIARY TRANSFORMER . . 18 4.1.1.3.3 RESTORATION OF POWER . . . . . . . . . . . . . . 19 4.1.1.4 FAILURE HISTORY . . . . . . . . . . . . . . . . . 21 4.1.1.5 CHANGES AND IMPROVEMENTS . . . ......... 21 4.1.1.6 CONCLUSIONS AND RECOMMENDATIONS . . . . . . . . . . . 22
I _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - --------J
_ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ _
.
2 ll Page 4. PLANT ELECTRICAL AND PROTECTION SYSTEMS . . . . . . . . . 23 4.1.2.1 SYSTEM DESCRIPTION - 4.16 kV ELECTRICAL SYSTEM . . . . 23 , 4.1.2.2 SYSTEM DESCRIPTION - REACTOR PROTECTION SYSTEM AND PRIMARY CONTAINMENT ISOLATION SYSTEM , . . . . . . . 24 4.1.2.3 OPERATIONS AND FAILURES DURING EVENT . . . . . . . . . 27 o 4.1.2.4 CONCLUSIONS AND RECOMMENDATIONS . . . . . . . . . . . . 28 4.2 EMERGENCY DIESEL GENERATORS . . . . . . . . . . . . . . . . . . 29 4.2.1 SYSTEM DESCRIPTION . . . . . . . . . . . . . . . . . . . . 29 4.2.2 OPERATIONS AND FAILURES DURING EVENT . . . . . . . . . . . 30 4.2.2.1 CURRENT TRANSFORMER . . . . . . . . . . . . . . . . . . 30 4.2.2.2 MECHANICAL PROBLEMS . . . . . . . . . . . . . . . . . . 32 4.2.3 CONCLUSIONS AND RECOMMENDATIONS . . . . . . . . . . . . . . 34 4.3 INSTRUMENT AND SERVICE AIR SYSTEMS . . . . . ..........35 4.3.1 SYSTEM DESCRIPTION . . . . . . . . . . . . . . . . . . . . 35 4.3.2 OPERATIONS AND FAILURES DURING EVENT . . . . . . . . . . . . 37 4.3.3 AIR SYSTEM RECOVERY . . . . . . . . . . . . . . . . . . . 41 4.3.4 CHANGES AND IMPROVEMENTS . . . . . . . . . . . . . . . . . . 44 4.3.5 CONCLUSIONS AND RECOMMENDATIONS . . . . . . . . . . . . . . . 46 5.0 HUMAN PERFORMANCE . . . . . . . . . . . . . . . . . . . . . . . 47 5.1 ORGANIZATION . . . . ........ . . . . . . . . . . . . 47 5.2 EVENT CLASSIFICATION AND NOTIFICATION . . . . . . . . . . . . . 50
,
5.3 USE OF SUPPORT ORGANIZATIONS AND PERSONNEL . . . . . . . . . . 51
!
5.4 MANAGEMENT ACTIVITIES AND COMMUNICATIONS . . . ... . . . 54 j 5.5 OPERATOR ACTIONS . . . . . . . . . . . . . . .... . . . 57 i
,
I l -
:( -
9 ., l "
'.i "
y 7, , ,. g iy , y -
;b w, *- (9 Asl ' ,
y , p . 1 i l' t a * a \* .
* Pace g
,
.i l
.
y (( 5.6 PROCEDURE ADEQUACY . . . . . . ! .g ............. 57 r - 5.7 RADIOLOGICAL ASPECTS . . . . . . s . . . . . . . . . . . . . .. 57
,
.5.8 CONCLUSIONS AND RECOMMENDATIONS . . ............. 59
${6.0SAFETYASSESSMENT................
.p-
. .. . . . . 60 j
. 60 g - [, 6.1yREACTOR SAFETY SIGNIFICANCE OF EVENT . . . . . . . . . . ^..y
, f.
p/ .
}s
.. i
. ,,/ '6.2 REACTOR SAFETY. IMPLICATIONS AT HIGHER DECAY HEAT LEVELS . . . . 61
>l' 1 6.3 STATION BLACK 0UT SAFETY IMPLICATIONS . . . . . . . . . . . . . 63
, 6.4 CONCLUSIONS AND RECOMMENDATIONS . . . . . . . . . . . . . . . . 65 ,
t - APPENDICES ~
\
'
APPENDIX A: ENTRANCE INT GVIEW ATTENDEES f
'
APPENDIX B: EXIT 0!NTERVIEW TTENDEES e , APPENDIX C: PERSONS INTERVIEWED i m ,
' -fn* s'
,. APPENDIX D: INTEGRATED SEQUENCE OF EVENTS U n .
APPENDIX E: BECO RECOVERY SCHEDULES g, APPENDIX F: INSTRUMENT AIR SYSTEM INTERRELATIONSHIPS
'
f APPENDIX G: INSTRUMENT AIR SEQUENCE OF EVENTS
*
APPENDIX H: ACRONYMS AND INITIALISMS t k'q l
.
' hz
,
l % i y' ' , ',' .,-
,#\ I
'f 4; .s .c .,
, 1 .
.s b
,
l4
'k t-f /
.
- ,, -_ -. -- -. - - - -_ - - . - - -
. - - _ - - - _ _ - - -
,, ' > ,
t Ll ', <
;1 1 ' t-l c ' [[ , ;
1.0 Erd g tive Summary
The ' Pilgrim Station experienced a tote.! loss of offsite power on November 12,1987, at 2:05 a.m. which extended for a period of 21 hours. The loss of offsite power was initiated as the result of a severe winter storm
, , consisting of gale force winds and wet snow. The winds whipped trans-mission lines into proximity to each other and snow was packed into high voltage switchyard insuiators causing phase te phase and phase to ground faults on the, offsite transmission line This resylted in the 345 kV ; switchyard neeekers open wg and i sola' ting the station. Both emergency i diesel generators started and ' assumed loads from the emergency buses, i Operator' actions whre correctly taken and the plant was stabilized and
}. maintained!i n a safe condition throughout the recovery perio In conjunction with the switchyard breakers tripping, the station startup traufermer experienced a phase B differential lockout trip. This trans-forrrer is the preferred power source fer outage conditions . and was the only source available for quick recovery of offsite power. Because of the phase B differential lockout trip, the decision was made to perform a series of tests on the transformer to verify no internal fault existe This was a major factor in the duration of the loss of offsite powe y At approximately, 11:30 a m. of that same day, a current transformer instru:nentatim circuit for one of the diesel generators was determined to have an open circuit and the decision was made to secure the associated diesel to preclude possible ' revere damag Once the diesel was secured, additional . equipment fa t iures and malfunctions delayed returning the diesel to operable status until 11:15 p.m. on Nov mber 14, 198 The station 'was in an 6 tended outage at the time and reactor safety was never a factw ' due to the viry low decay heat. Hcceve , the plant had equipment cut J service and was configured such %at, under other circumstances R n higher decay heat loads, it couM nave created serious operations.1 inflexibilitie Fce example, the 23k\' shutdown transformer, an alternate off site power source, was out of service because of plant modifications and maintenance. Alsc, two of the three emergency bus supplied instrument air co'npressors were out of service for maintenance wor Once the diesel was secured, the only running instrument air compressor stopped because it was powered from that diese The other alternate method of supplying power to the station was through backfeeding of the main and the unit auxiliary transfcmers. However, this j
,
method was not readily available due to the need to remove bus links ' be ween the main generator and the transformers. Ultimately, this was the method used to restore offsite power to the sit The operational staff resperded well to the event and adequately coped I with the equipe nt failures and malfunction One miscommunication between the on-watch supervisors prematurely secured the diesel generator, but this had no adverse safety effects on the st n ion . The recovery
efforts were well planned and reasonably well implemented .once the i _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ - _ - _ _ ._ _ _ _ _ - - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ - - _ _ _ _ _ - _ _ _ - _ _ _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ i.
I
L conditions were properly assessed by station managemen Delays in restoring the diesel to operable status were caused by inadequate maintenance practices before the even Because of the frequency with which the Pilgrim Station has experienced losses of offsite power, the complications which resulted during the recovery efforts and the extended period the plant was without offsite power, an NRC Augmented Inspection Team (AIT) review of the event was performed during the period November 16-20,198 Although there were no violations of regulatory requirements identified, specific weaknesses were noted that warrant additional licensee and NRC attentio These are maintenance procedure inadequacies, operator alarm response deficiencies, and equipment malfunctions (see Section 2.3).
Other observations regarding human and equipment performance were made and are referred to the licensee for their consideratio .0 Introduction 2.1 Scope of, Inspection In response to the loss of offsite power (LOOP) event at the Pilgrim Nuclear Power Station during the early hours of November 12, 1987, the NRC formed an AIT to determine the sequence, the causes and the safety significance of the event. This was ~ accomplished by establishing a chronology of the event, reviewing equipment performance and plant staff actions relative to the plant design bases, technical specifications and operating procedure The NRC team held an initial planning meeting on November 15, 1987, and arrived onsite and performed the inspection during the period of November 16-20, 1987. Attendees at the entrance and exit interviews are listed in Appendices A and B. Individuals formally interviewed by the NRC team during the course of the inspection are listed in Appendix .2 Team Composition The team was composed of a team leader a1d seven headquarters and regional specialists with expertise in plant operations, electrical engineering, instrumentation and controls, risk and safety assessment, and management control .3 Summary of Recommendations The consensus of the AIT staff regarding the station's response to the loss of offsite power was that the operating staff performed well and that the reactor plant safety was never in questio The actions taken for recovery were appropriate and well organized. Coordination and communication between groups was acceptable but improvements are needed to increase , effectiveness. The coordination of these groups would be substantially enhanced by well defined management guidelines for this type of event i _ _ _ _ _ _ _ _ _
- _-__ -
.
(i.e. one in which the formal emergency response organization is not mobilized).
The recommendations presented here are the result of observations made by l the AIT and are not violations of regulatory requirements but do l represent concerns that warrant licensee review. This is a summary of l significant recommendations.
Sections 4.1.1.6 and 4.1. The operators were not aware of the alarm indicating the reduced voltage on the 345 kV offsite power source prior to the loss of offsite powe They were also unaware of the alarm indicating the blown fuses in the analog trip system power supply. The failure to utilize these alarms should be reviewed and appropriate corrective actions developed.
l' Section 4.1. The operation of the startup transformer differential lockout relay was apparently the result of a transient for which the protection was not designed. The transformer did not experience an internal fault and the operation of the lockout delayed the re-energization of the station from-offsite power sources. The actual cause of the differential lockout needs j to be conclusively establishe Section 4.1. The blown fuses in the analog trip system were the apparent result of a common cause. The cause of this condition should be identified and corrected or determined to be acceptable before the reactor is restarte Section 4. The inoperability of the "B" emergency diese.1 generator during the event resulted from inadequate or incomplete maintenance procedures. The binding of the prelubrication pump and the leaking fuel injectors could have been prevented from interfering with the recovery operations if adequate procedures for repair and post maintenance testing were employe Section The plant configuration before the event and the equipment that was out of service for maintenance purpose created operational situations that could have been more serious under other circumstances with substantial decay heat. Describe what considerations will be made in the future to assure that essential and non-essential equipment removed from service for outage maintenance do not create undue operational inflexibilitie .0 Event Description The Boston area experienced a storm beginning about 5:30 p.m. on November 11, 1987, and continuing through the night until about 10:30 a.m. on the 12th of November. The National Weather Service characterized the condi- - _ - _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -
- _ _ - _ - _ _ - ._
..
tions as gale force winds and wet sno For a three hour period during the storm, blizzard conditions existed (See' Table No. 1). The site ; meteorological tower records for the 220' elevation show that wind speeds averaged in' excess of 50 miles per hour during the period of interest, 1:00 to 3:00 a.m. on the morning of November 12, 1987, with gusts peaking to 60 miles per hour (Figure Nos.1 and 2).
When the event occurred, the Pilgrim Nuclear Power Station reactor was shutdown in an extended outage which began in April 198 The reactor mode switch was in the " refueling" position and reactor coolant tempera- ' ture was being maintained between 170 to 190 degrees F using the residual heat removal pumps. Temperature was being maintained at this level in preparation for a reactor coolant system hydrostatic tes The station offsite power sources are two 345 kV lines, No. 342 and No. 355, which feed power through either the startup transformer or the auxiliary transformer (see Figure No. 3). In order for the auxiliary transformer to supply power to the station during an outage, the isophase ; bus links to the main generator must be remove This arrangement is referred to' as "backscuttle" by the licensee. The station can also be supplied by a 23 kV shutdown transformer; however, this source was unavailable for use because preventiva maintenance was in progress and the system was being modified to install a new, third diesel generato At 2:05:26 a.m. , line No. 342 experienced a phase "A" to phase "C" to ground fault. Because of the ground fault, relay operation opened breaker Nos.103 and 104 at the Pilgrim Station. Breaker No.104 was slow in opening which resulted in the stuck breaker protection operating to open breaker No.105 and sending a transfer trip signal to Auburn Street, opening breaker No. 2130, and Canal Station, opening breaker No. 31 Breaker No. 412 was previously open before this even Line No. 355 was still energized and powering the startup transformer through breaker No. 10 At 2:05:34 a.m., line No. 355 had a phase "B" to "C" faul Breaker No. 1670 at Bridgewater opened and automatically reclosed 0.5 seconds .j later. In conjunction with breaker No. 1670 opening, breaker No. 102 i supplying the Pilgrim Station startup transformer opened on a transformer phase differentia During the 0.5 second time required for reclosing breaker No. 1670, the licensee hypothesized that motors, normally energized from the 345 kV transmission lines through the startup transformer and the Pilgrim distribution system, generated a voltage backfeed through the startup transformer to the fault on the No. 355 lin This backfeed caused the motors to quickly slow down with a resultant frequency deca The i frequency decay resulted in an increased voltage / frequency ratio (volts per Hertz overexcitation) to the startup transformer which caused the HU-1 differential protective relaying on the startup transform 2r to activat Actuation of differential relaying caused the breaker Nos. 102, 152-104, 152-204,152-304,152-404,152-504, and 152-604 (See Figure No. 4) to trip ope __________________________-______-____
- _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ _ _ _ - - _ - _ _ _ _ _ - _ _ _ _ - _ _ _ _ . 5~ TABLE NO. 1 BOSTON AREA WEATHER CONDITIONS NOVEMBER 12, 1987 WIND WIND LIQUID TIME DIRECTION SPEED GUST PRECI (Degrees) (Knots) (Knots) (IN) 12:50 .06 1:18 :54 .11 2:25 :54 .13 3.35 :53 .12 4:54 .16 5:54 .12 6:54 .10 7:53 .05 Provided by: National Weather Service
!
I ___ _____ ____ ____________________ ____________ -
e
[
I
,
, *N
%)] }%X J',e
" * re .
*
,,g l p
,
,
,
-
, .. i h'$ ;"$. ,,
""a g l b'
4.t : I i /l3 ih' ' fi m ". l
'
;
'liSiTNt! "'"'
,7
'
7
-
.; M'.-sds l
}l0 lI(I 8h[ 31 $ {l b 23 O " ) 0)
.. ... ;
.
t n fp*.t:p ..;,,,,. i. p :
. .
. ,1
'
g
-"
RiRI .' o
]
'
.,A
? .l 'I'
~
..
-
"fQ iNNN;N,N_,. didi U ,,
',i ....
gh *
.
.
,
.
..
. .
' gig;.ph :t EN L
~ ..
.
-
; .
-
s.w.,g; ,,, .y p
.. - ...
. , . .. . ,
'
h, ,,$ i
'. &
,
n' * , . hgY / O m
, C< c- \ , , ,
n .r . h',h ... 5 O l l,
' ' ? - - '
i f '"5 iL i) in$pg.Rl
. ; ' , :
,. . . hf . ;
'
'
'
P.TO Bl0 33
'
31 9 ' 23 C
"
E b
}},( ; .
.
-
.. .
'
T 1. n.p,'
.
W :.2,
-
ta -
.
<
n;
,
.: . . p r
.,, I
. i
'p -
,(r
< - n i
,
. ,, : ..,. J
.n . o
- '
.' *
\
i ll .=
/,h k dh?
~
'
, ,i
" j
:,
-
'
.
-
.
: -
9%%dl ,
'
,
a . .
-
ph%yl- .. _
-
.
.
. .. s
. . d .
i /. h.n.,.,N .
"RE u
\.
- a
.
-
,
d* gun
'
s
; m - I .
..
-
- -
;
>
.
WifG: uk '.L . hf;.h ,
-
s
-
-
t ,.
'
r et.. I l
' '
,
,
-
'
IhcNkh Ni h! QQWik[)
.
? $ \MITD usiG II ' !
Ly"j?m ._
.
jlt ggna -i; '
,a n.!
, Mam > ,i:: a, , o .
. j,,g <ca ; ,,
.
,
i l
~
W
'
'
l -
, r i ' {p /h.A,1. p' !: '
, .s 2 . fST' 'I',
@
'- 1
~ -
$. [ **fG,h i- . i; p
.{
.
'
bFIGUREN WIND SPEED AND DIRECTION k g l * l 2 ', , N q
! !
O,
, .
, '
' 223 FT ELEVATION [d . W ,w'h,v;..- e - -
>
n
; q, des. *, I
.
I
,, '
I
[ )' " '
i h .. . l ; I r N!.: efi ! I l h?N
..
,
'
.
h, . ! l uM ,hb o
, -p- . . . . . . . . . . . . . . . . . .
.
. . . . . . . . .} n?Q.s. % .
1: _
'
-
h( l men =l pjf,[ff.' e- ,H il g}$:gA.M@
'
I 11I!' '
'9 '
,,;
I ld ~ lllM H M l,
-
1 Ill ' i i e. 1 o . .
~
; .* . , .
'
'
'
T.,fIl _ !'
{
,,
~
-
' '
.
'
,
-
yhM., '
.
@
!ll ;[(.I$.2 "U: f;- --+
-
- ~" - -
i
-?{
- -
.
N+
.
,
._.._. , .,,
~$ . ,
)1
.
.f .
. 1
'
,
. .
[
, Ia s M;,;4II
'
j(l' i , ZD 81[d '3) I', .. ,Fj l iB )
'
' " ' " '" " " "' ~ " "'~
'
,
l } l l ,
'
s
-
- - . .I
, _ ? : ...;
- -
'
. -
l4<
'* n if ; ,
,
.; : ;p
'
-: E ll;7
}
-
"
_, y p
'
' '
2I
'
IM.',l_ 5 '. -[5M l Ili~']l:
-
< . , i
.- ;
,
g -
-
u,
-
3' : . o
-
.
;;:
-
. . o o ,
-
..
#: ,
-
'
r
'
1 ,
-
,
-
n; no ; , . i __ _
'
.
.1:.; '
t-
: - 'j ll;;;
~
'
- ' '
l y .- <
: j
- .
a ., --
.
, n ...., .,
,
,.. . .
g ,. , - <., ..,.... . a I g .. ' 7)
"
-
c *
.
, l_ n i
.
,
3
'
. A
'
, --: -- * O L':
'
'
ssfj/r' D
' ,
s i "=' L",' il; y,iDg;p-, y,:> I
'- '
. , L ,. 6 .14
;..- A N >1'
., ,
f
'
'
m ;. . 'y
-
;, ,
P r .,, ,p- /S4 .f l 1 L * '
;
'
'
Til
'
hIl dDl , 80 M I,, j JL 'L ::: 4;i E] B3 o _
-
., ,,
" '
t 1 Qg :,ty- 59WC
-
,
r 1 ;
.e
-
-).di .- !
!
' '
(
- ~
i : i
"'Y ~J"'r
.
'
j. lf'I' ,, ; .
- - FIGURE N h-% 5' ' - - - '- ""-~~ - --
r-
{".=- 'I WIND' SPEED AND DIRECTIO:1 33 FT ELEVATION f- ((
i k , l %..
, l % 'd' * "" '" * *
e, ;.g . ,
' '
m t a na> h, .y , i; 49 - -
..
t
, as;,12ar _ .
F .. .- - ..
'
'
.
f; . h e m en
,
wmm ,
.:: I a n..% - - - _ - - - - - - - - - - - - - - - - - - - - - - - - _ - - - - - - - - - - - _ - - - - ---
_ _ . _ _ _r e To _ . _
,r,e
_ . .
,f ,
,
,
'
T
*
7 f b
,o * r c c ,7 2_ 2 C e e a ct no n2 1 44 ee L3 0N '
u o nA n a s8 n ] 0= s4 e 1e e0 04 s 40 s - Dr s
-. To u s/
s O co At 0/ a 8 J E A L M 7 1J % % , Jm E
(N S*
A3
%
.Y r a~ g R o E n M
At K c Cna4 t r8 re A f o s 's 3 h1* es N e R e "i
" G -
E n h R s. 6A
* i *
CO n c- .
,
A 'a
' At o $ *
M J D' e 3 c 6 Af
-
sr 's s s A am , O os M uAL
.
Ag e, r ea t A * e
-
nA l M,,
%q u 0z q c s ce AI r oi or o r/
e s i Dp
,
rs r
,
gg a a n3 ro Sr s1 r-
-
D aT
%
L
a=6 T-C rc E- V 4 7 R t1 e k M/6 w s o n 5 f Tg5 o9 n8 4 A3 c- a c/
W3 D sT i Se N O
/
K2 0 3T I Gn a A n .TD K OSR B N MY EI H RRC UGT GLI I I FPS
, ,l 1
, w
' . %
'
-
-
_ a
-
n
-
.
+
,
[#
k -
-
-
q ,- - p I) -
% g n ,
V.- * f' a g 85 E
' 4 w./v ' -' .
l r
I
)l S'93 g
g g y *, 3) , f* k
.
t t p V ra [' , K l
-
A l h -
'
V *
y l
$"j
)
A
* I E'32 A
g
% in '
)
G S ) , u E S h<r tt e s U fe
/
B c A L o. D % , I v C I k, g g-R - T , 2, g 4C
.E g sg s m a~.
OE N Ek L V nR eG g
'
)
y alea . R AH gy
,. g U6 )
G1 I F4 s R ys aF
,<
) , -
h
G A$ g 19 57r re dA tC t u l f U ~Y f c A l
/ $
v. 0 0 y iv g S6-R -
! )i ,
'*.
Es e
,
y n v
-
s8 g* f f I
) 3 - .
.
t - n _ m _ a f s s_ vi
-
v
- .
J, l l c,. N :s " a v kR M
, km U7 ts e sI )
'
r b' m in s a O7* m, a t aw ' 3 s Uu G I)1I1 l' ll l
_ _ - _ _ _ - _ - _ . _ ._ _
1 The switchyard and .offsite breaker operations resulted in the station ! losing all offsite power and the emergency diesel generators starting to j supply the onsite emergency buse All plant systems responded as designed including a full reactor scram and a primary and secondary i containment isolation. However, efforts to restore the shutdown cooling l after the event by starting the residual heat removal pump resulted in i another primary containment isolation. Subsequent efforts to re-establish shutdown cooling were successfu Shutdown cooling was not a safety concern at this time because of the very low level of decay hea The licensee assessed the event as reportable pursuant to 10 CFR 50.72 and notified the NRC at 2:55 As the duration of the loss of offsite power continued into the morning of November 12,'the plant operators experienced additional equipment failures which required operator intervention. At 5:55 a.m. the operators identi-fied blown fuses in the analog trip system panel; three of the four panels l had blown fuse At about 6:30 a.m., the plant began to experience instrument air problems because the normal sources of compressed air are powered from nonvital electrical buses. A single air compressor, powered by the "B" diesel generator, was not able to cope with the deman The other two air compressors that are powered by emergency sources were unavailable because of maintenance work that was required to restore them to service.
l At about 10:00 a.m. the plant staff noted that the "C" phase ammeter of the "B" diesel generator was reading zero. This resulted in a recommenda-tion by plant technical personnel with concurrence from BECo Nuclear Engineering Department personnel to secure the "B" diesel to prevent further damage. The concern was that an open circuit existed which could l impress excessively high voltages on the circuit component The "B" diesel output breaker was opened at 11:30 a.m., but reclosed automatically due to the presence of a LOOP signal. An operator was dispatched to the diesel generator building and manually secured the diesel by using the emergency keylock switch at 11:35 The securing of the "B" diesel generator left the plant with only one source of power, the "A" diesel generator. Also, it removed the power source for the only remaining air compressor. The loss of all instrument air, although a nuisance, presented no safety hazard to the plan The plant management mounted an organized recovery effort employing multiple approaches. The recovery of offsite power involved attempts to restore the startup transformer, provide power through the auxiliary transformer and an assessment of energizing the 23 kV shutdown trans-forme In order to support the recovery ef fort, the Nuclear Department Operations Manager administratively staffed the technical support center at about midda This allowed the operating shift to focus on plant condition Before the startup transformer could be safely re-energized, it was determined that several tests should be performed to verify no internal fault existed. Complete washing of the high voltage insulators within the switchyard was also thought to be necessary to ensure that fault tripping _ ______________ ___________ __-
- _ _ _ _ _ _ .
a problems would not occur. The testing involved Doble Testing of the high voltage side of the transformer, insulation resistance testing of the low voltage side and oil sample analysi These tests and cleaning of the insulators were completed at about 2:00 a.m., November 13, 198 In parallel to the efforts to restore the startup transformer, actions were initiated to restore power through the a'uxiliary transformer (back-scuttle). This also required the washing of the high voltage insulators in the 345 kV switchyard in addition to the removal of the isophase bus quick disconnect links between the main generator and the main trans- ; former. This was accomplished and offsite power was first restored by backfeeding through the auxiliary and main transformer at 11:09 p.m., November 12, 198 Restoration of power through the 23 kV shutdown transformer was determined not to be a viable method of restoring power to the station due to the extent of work necessar This method was not pursue An integrated chronological sequence of events is presented in Appendix l .0 Equipment Performance 4.1 Electrical Systems 4. kV Electrical Offsite Power Source 4.1. System Description 345 kV Switchyard: The offsite power distribution system consists of a 345 kV ring bus and its associated line connections (see Figure No. 3). ' The ring bus has tie lines to the Canal and Auburn Street (No. 342) and Bridgewater (No. 355) lines. These incoming transmission lines run adjacent to each other on the same tower for approximately eight miles and then diverge at the Snake Hill Road Tap. The No. 355 line runs westerly and northwesterly approximately twenty-one miles to the Montaup Electric l' Company's Bridgewater Station. A tap is made to the No. 342 line at Jordan Road approximately five miles from Pilgrim Station. This line runs approximately twenty-six miles northwesterly to the Auburn Street Station of Montaup Electric Company. The other branch of the No. 342 line runs southerly for about thirteen miles to the Canal Station of Commonwealth Electric Compan The tie-ins to the 345 kV ring bus occur via manually operated disconnect switches. The unit main transformer (X1) ties into one side of the ring bus via manually operated disconnects and the start-up auxiliary trans-former (X4) taps off the opposite side of the ring bus. These connections are isolated by 3,000 amp, 345 kV, air blast circuit breakers (ACB).
The 345 kV transmission lines and ring bus are arranged so that failure of either line will not result in the loss of the main generator, the other 345 kV line, or the startup transformer. The transmission system is protected using carrier relaying on the lines and high speed differential . _ _ _ _ _ _ _ _ - - - _ _ - _ _ - _
- _ - _ _ _ _ _ _ _ _ _ ___
protection on the transformer Overcurrent, ground differential and impedance relays protect station components within the ring and protect the grid from faults within the ring. A fault detected by switchyard y otective relaying will trip breakers adjacent to the fault as long as the bus is energize The 345 kV breakers are controlled directly from the main control room. Breaker position, 345 kV line voltages and other parameters are monitored in the main control roo Unit Alternating Current (AC) Power Source: The station main generator provides power through the isolated phase bus at 24 kV to both the main transformer (XI) and the unit auxiliary transformer (X3). The generator voltage is stepped up through the main transformer to the 345 kV ring bus voltage. The generator voltage is stepped down to 4.16 kV through the unit auxiliary transformer for the station auxiliary power distribution system (see Figure No. 4). The unit auxiliary transformer has two 4.16 kV windings, X and The X winding feeds 4.16 kV switchgear buses A3 through A6 while the Y winding feeds 4.16 kV buses Al and A Refer to paragraph 4.1.2.1. for details on the 4.16 kV syste Preferred Alternating Current Power Source: The preferred ac power source (startup transformer) provides a source of offsite ac power to the auxiliary power distribution system which is adequate for the startup, operation and shutdown of the statio The startup transformer (X4) supplies power to the station auxiliary power distribution system whenever the main generator is offlin After the main generator has been synchronized to the 345 kV system and has been partially loaded, the auxiliary power distribution system is manually transferred from the startup transformer to the unit ac power sourc Automatic fast transfer capability is provided to restore the preferred ac power source to the station auxiliary power distribution system in the event that the unit ac power source is lost for any reaso Should power be interrupted to the preferred ac power source due to a double 345 kV line fault, it will be automatically rest %j when the line breakers reclose after the fault is cleared and the lines are re-energized. This automatic reclosure is designed to prevent both 345 kV breakers from reclosing at the same tim The breaker control for the preferred ac power . source is interlocked to prevent interconnection with the secondary ac power source. The preferred ac power source may be synchronized and interconnected with the unit ac power to permit live source transfer following synchronization of the main generato .1. Operations and Failures During Event At the time of the initial loss of off site power on November 12, 1987, the 345 kV ring bus was fully energized with all breakers closed. The startup transformer was supplying all 4.16 kV buses and both diesels were in standby. The main transformer was isolated at the ring bus disconnect switch, T-930, and the main generator iso phase disconnect links were closed. All 4.16 kV buses were isolated from the unit source. The 23 kV secondary source from the shutdown transformer was not available due to _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .
_ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -
l
maintenance which included the tie-in of the new, third " blackout" diesel generato At 1:27 a.m. on November 12, a 345 kV line low voltage was alarmed on the l main control room computer logger. Analog voltage indication was 338.73 kV. The value remained below the alarm level of 341.50 kV until all power ' was lost at 2.06 a.m. The nuclear operations supervisor (N05) and nuclear . watch engineer (NWE) indicated that they were not aware of the alarm. It i is not' clear whether any other operator was aware of the low voltage I condition; however, no actions were taken to initiate procedure. 2.4.144
" Degraded Grid Voltage" which would have included monitoring 345 kV bus voltages and contacting Rhode Island, Eastern Massachusetts, Vermont
! Energy Control (REMVEC) about grid stability. More specifically, a check of the 345 kV bus voltage recorder in the control room clearly shows the i above mentioned voltage drop which was sustained until the complete loss of offsite power. Although in this case missing the alarm had no impact on the overall sequence of events, the failure of the operators to consult the appropriate procedures in response to the computer alarm .is of concern to the NR At 2:05:26 a.m. on November 12, 1987, one of two 345 kV transmission lines (No. 342) connecting the Pilgrim Station switchyard to the Boston Edison transmission network, experienced an "A" phase to "C" phase to ground fault (see Figure No. 3). As a result of this fault on the No. 342 line, protective relaying operated causing breakers No.103 and No.104 to trip open to isolate the fault. Also, because No. 104 breaker was indicated slow to open, backup or stuck breaker protective relaying operated causing a transfer trip signal to be sent to breaker Nos. 103, 105, 2130, 412, and 312. The transfer trip signal caused breaker Nos. 105, 2130, and 312 to trip open and caused the automatic reclose feature on breaker Nos. 103, 105, 2130, and 312 to be removed or locked ou For a short time following isolation of the initial fault on the 342 line described above, the Pilgrim Station was energized from the remaining 345 kV transmission line (No. 355) through breaker 102 and the startup transformer. At 2:05:34 a.m. (7 seconds following the initial fault on the 342 line) the 355 line experienced a "B" phase to "C" phase fault. As a result of this fault on the No. 355 line, protective relaying operated causing breaker 1670 to trip open and to reclose 0.5 seconds later, i During the 0.5 second time required for reclosing breaker No. 1670, the licensee hypothesized that motors, normally energized from the 345 kV transmission lines through the startup transformer and the Pilgrim distribution system, generated a voltage backfeed through the startup transformer to the fault on the No. 355 line. This backfeed caused the motors to quickly slow down with a resultant frequency deca The frequency decay resulted in an increased voltage / frequency ratio (volts per Hertz overexcitation) to the startup transformer which caused the HU-1 , differential protective relaying on the startup transformer to activat Actuation of differential relaying caused the trip opening of breaker No , 152-104, 152-204, 152-304, 152-404, 152-504, and 152-604 and the loss of offsite power to the Pilgrim station from the Boston Edison transmis-sion networ _ _ _ - . - - - - - _ _ _ _ _ _ _ _ ___j
_ _ _ _ - - _ _ _
-- -__
4.1.1. Startup Tra . 'f; ,ner The station lost all ot 7 te power at 2:06 a.m. when breaker No.102 opened. This was due to i,ie actuation of the B phase differential relay
<
of the Startup Transforme The Startup Transformer is the immediate access for offsite power to the station onsite electric distribution system, i.e., the preferred power source. The transformer is a three winding, three phase, 345-4.16-4.16 KV unit, with the primary side (H winding) rated 20-26.6-33.3 MVA and each secondary (X and Y) rated 10-13.3-16.65 MVA, OA/FA/FOA. .It is located in the switchyard at the Southern Boundary. It is connected, on the primary side', to the 345 KV switchyard ring bus between ACB 102 and ACB 103. The transformer secondary sides are connected to the station 4.16 kV buses through underground cables, the X winding supplying buses A3, A4, A5, and A6, and the Y winding supplying buses Al and A2. (see Figure No. 3.) The transformer is designed to supply the power requirements of the station's two emergency service buses A5 and A6 (feeding safety-related loads), and the four normal service buses A1, A2, A3, and A The Startup Transformer is used to supply the 4.16 kV buses during normal plant startup and shutdown. After the main generator has been synchron-ized to the 345 kV system and loaded, the 4.16 kV buses are individually transferred from the Startup Transformer to the Unit Auxiliary Trans-former. In the event the Unit Auxiliary Transformer is lost, the 4.16 kV buses are automatically transferred to the Startup Transformer, the pre-ferred power sourc Prior to the operation of the startup transformer differential relay, the transformer was supplying all the required electrical loads of the station 4.16 kV and 480 V systems. When the B phase differential relay (187-48) actuated it energized the startup transformer lockout relays (186-4 and 186-4X), which opened switchyard ACB No. 102 and all the 4.16 kV startup transformer feeder breakers to buses A1, A2, A3, A4, AS, and A6. The relays also actuated an alarm on panel C-3 in the control room. On loss of power to buses A5 and A6, Diesel Generator A and Diesel Generator B automatically started and energized the respective buses as designe Differential protection of the startup transformer is provided by three single phase Westinghouse HU-1 transformer differential relays (designated 187-4A, B, andC). These relays provide protection for faults internal to the differential zone extending from the primary side bushings of the transformer to the 4.16 kV startup transformer feeder breakers at buses Al through A6. The differential relays are not designed to operate for faults external to the zone; neither do the relays operate on magnetizing in-rush currents associated with energization of the transforme For moderate internal faults, the differential unit and harmonic restraint unit of the differential relay will operate tripping the relay. Operation of these two units, in turn, energizes the indicating contactor switch (ICS) unit which completes the trip circuit through the transformer lock- - _ _ _ _ _ _ _ - _ _
_ ___ _ _ _ _ _ _ _ - _ _ - _ _ _ _ _ _ - -___ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \
out relay. An indicator target on the HU-1 relay (the right target, when viewed from the front) will also dro For heavy internal faults, the indicating instantaneous trip (IIT) unit will operate and complete the trip circuit to the lockout rela In this instance the left indicator target will dro During the initial stages of the Pilgrim LOOP, startup transformer differential relay for phase B actuated, alarmed in the control room, tripped ACB No. 102 and caused the loss of power to the statio The control room operators recall a distinct time lapse between the opening of ACB No's. 103, 104, 105 and the opening of ACB 102 (due to the operation of the differential re'l ay) . The right indicator target on the differential relay was noted by the operator, indicating ICS operation. After washing and cleaning the transformer, the primary and secondary sides, including the 4.16 kV cables and breakers were tested. The transformer oil was also tested. The transformer passed all tests and, on November 14, 1987, the transformer successfully supplied power to the station. Based on success-ful tests and operation of the transformer, it can be concluded that either the relay misoperated or spuriously operated for a fault outside the zone of protectio An analysis by the licensee on the relay operations data at Pilgrim and other substations on the 345 kV system during the snowstorm of November 12, 1987, presented the following reason for the transformer differential relay operation: The motors at Pilgrim Station, normally energized from the startup transformer generated a voltage backfeed to line No. 35 The motors quickly slowed down with a resultant frequency decay. This resulted in an increased voltz per-hertz to the startup transforme As a result, the HU-1 transformer differential relay operated to open breaker No. 10 The analysis done was based on an assumption that the transformer differential relay actuated after the 2:05:34 a.m., line No. 355, phase B to C faul Based on the above data, the transformer differential and the subsequent opening of ACB 102 could have occurred before the fault on line No. 35 The differential could have operated on a fault on the high voltage bushing of the transforme A previous event in which the startup transformer phase C differential relay actuated for a fault on the high voltage side is described in Pilgrim LER 83-04 .1.1.2.2 Stuck Breaker The control and protection logic for each breaker in the ring bus at the switchyard is designed such that if a breaker receives a trip signal and does not open or is slow in opening, the stuck breaker protection scheme comes into actio This feature will trip the two adjacent breakers in _ _ - __
- - _ _ _ _ -_ __ _ . _ _ _ _ _ _ _ _ . _ _ _ __
the switchyard thus protecting the ring bus and the grid from faults not initially isolated. The scheme, in addition, sends a transfer trip signal to adjacent breakers at the other end of the transmission lines at Canal, Auburn or Bridgewater stations to further isolate the initial faul At 2:06 a.m. on November 12,1987, the 345 kV line No. 342 had a phase A to phase C to ground f ault, which through protective relaying opened ACB No. 103 and ACB No.104 at the switchyard (See Figure No. 3). However, because of the indicated slow response of ACB No.104, .the stuck breaker protection operated, as designed, to open ACB No. 105 at the Pilgrim switchyard and send a transfer trip signal to open breaker No. 2130 at Auburn Street station and breaker No. 312 at Canal Stree i
!
The licensee is investigating the root cause of the slow response of ACB 104 to determine if the problem is with the main breaker mechanism, the auxiliary contacts, or the associated relays, and whether the problem is related to the cold weather. Corrective actions based on the findings of the investigation should improve the reliability of switchyard breaker operatio .1.1. Swinging / Oscillating Transmission Lines At 2:06 a.m. on November 12,1987, the 342 line experienced an "A" phase to "C" phase to ground fault. Following this initial fault on the No. 342 line, the No. 355 line experienced a "B" to "C" phase fault. These faults on the Nos. 342 and 355 lines were believed to be caused by snow and win With snow buildup and dropping of snow at different times, the various phase lines and the static line would be swinging with the wind and oscillating at different rates causing them to come into proximity with each other. When the lines came near enough, a fault was detected by protective relaying which isolated the line .1.1. Insulators At 2:06 a.m. on November 12, 1987, the Nos. 342 and 355 lines were isolated by protective relaying due to faults on the line Following isolation of the Nos. 342 and 355 lines, the insulators located in the Pilgrim Station switchyard were observed to be packed with snow from top to bottom on the sides facing the northeast win Also, the insulator skirts were imbeded in snow. At this time with their snow covering, the insulators would have been unable to perform their design function of insulating the 345 kV lines from ground. The snow would have allowed, if energized, the 345 kV lines to fault to groun The insulators required to re-energized the startup transformer were washed, cleared of snow, and returned to operable status at 1:30 p.m. , approximately 11.5 hours following the initial isolation of the fault on the Nos. 342 and 355 lines. The insulators required to re-energize the main and unit auxiliary transformers for backfeed were washed, cleared of snow, and returned to operable status at 8:00 p.m., approximately 18 hours following the initial isolation of the fault on the Nos. 342 and 355 line __-____ _
g
4.1.1.3 Recovery Subsequent to th'e loss of offsite power at 2:06 a.m. , operations and electrical maintenance department's efforts involved investigating the cause of all switchyard breakers tripping open and the lockout of the-startup transforme It was recognized from the very beginning that washing of the switchyard would be required due to the severity of the storm and snow accumulations. The recovery activities involved with washing of the switchyard are described below. At the morning production meeting, which was chaired by the Outage Manager at 7:30 a.m. , the Modifi-cation Management Group was directed to determine the feasibility of returning the 23 kV shutdown transformer to servic By approximately 10:00 a.m. a recovery team was formed under the direction of the Outage Manager (Maintenance Section Manager). This team consisted of members of the following licensee groups: outage management, station management, corporate oversight, nuclear engineering department, administration, technical, and maintenance. Individuals were tasked with specific assign-ments, which included plant walkdowns to develop estimates of effort necessary and sequences needed to complete the various recovery option Where applicable, plant procedures were reviewed during the walkdowns. At this point, the licensee concluded that the return of the shutdown trans-former to service was not feasible, and no further consideration of this option was warrante The options being considered consisted of: 1) returning the startup transformer to service, and 2) backfeeding (back-scuttle) the switchyard through the main and unit auxiliary transformer However, at approximately 11:30 a.m., the "B" diesel generator was removed from service to investigate the loss of a "C" phase current transformer instrumentation circuit. Planning efforts to return the diesel generator to service were initiated and became one of the options to augment the "A"
*
diesel generator, which was the single source of power available to operate plant equipmen The results of the licensee's initial planning and coordination efforts for recovery of the plant from the loss of offsite power is depicted on the schedule of events released at 1:00 p.m. on November 12, 1987, which is enclosed in Appendix E of this report. A subsequent revision to the schedule was issued by outage management later in the day at 7:00 p.m. , and on November 16,1987 a schedule of the actual implementation of the recovery plans was release These documents are also included in Appendix As part of the recovery effort, the team leaders assigned responsibility for the various parallel options being implemented for power recovery were tasked with reporting their status on an hourly basis to the Technical Support Center. In addition, the Manpower Coordinator was tasked with the , responsibility of determining what personnel were onsite, their hours J worked, and the disciplines represente A sleeping area and meals were provided to workers involved in the recovery ef for The AIT concluded that the planning efforts conducted by the Outage Management Group were a positive factor in coordinating the recovery effor However, there appeared to be a lack of participation in the recovery planning process by the operations grou l _ - __ ___ _ _ ____ _
_ _ . _ _ _ _ __ _-
,
l 4.1.1.3.1 Washing of Switchyard i Directly following the loss of offsite power, the electrical supervisor on shift assessed the switchyard conditio Because of high winds and i driving snow conditions, activities were limited to those involved in I preparing for the hot washing of the switchyard. This action consisted of . moving the hot washing machine from the maintenance shop to the switch- l yard, and connecting the unit to water and power. As requested by the l Nuclear Watch Engineer (NWE) the Senior Electrical Engineer in the Main-tenance Group arrived onsite at 4:00 a.m. By 4:30 a.m. his walkdown of the switchyard determined that 1) the startup transformer had 3"-4" of heavily l packed snow that was bridging the gaps of the insulators; 2) the switch-l yard insulators for disconnects and ACB's were heavily coated with snow; 3) main and unit auxiliary transformers had snow buildup but did not , require washing at that time; 4) the switchyard would need selected l washing to insure reliable reconnection of offsite power; and 5) the pre-ferred method to restore offsite power to the station would involve j returning the startup transformer to servic The switchyard was washed in accordance with maintenance procedure M.3-20, Rev. 2, Live High Pressure Wet Washing Procedur Switchyard configuration at the initiation of washing consisted of lines No. 355 and No. 342 de-energized, all disconnects closed except T900 and T-930, and ACB's No.s 102 through 105 open. Although all preparation for washing was complete at 6:00 a.m., the required authorizations from the NWE and REMVEC were not provided until 6:53 a.m. The sequence of washing started on the line side of the T901 disconnect to the startup transformer bushing and 103A disconnect (see Figure No. 3). At approximately 1:30 p.m. this phase of the switchyard washing was complete, which allowed power to be restored to the 355 and 342 line Following the initial washing ef fort, the maintenance personnel moved the ' washing equipment to the other side of the switchyard to support the effort to restore power by backfeeding through the unit auxiliary trans- l former. This phase of the washing effort was initiated at approximately
'
5:00 p.m. and completed at 8:00 p.m. The equipment washed consisted of disconnects 105A, 105B, 104B, T931, and ACB 105. Throughout all washing i efforts, the equipment being washed was de-energized. A warming trend in temperature and reduction in wind strength resulted in the second phase washing efforts being accomplished in less time than the first phase (i.e., three hours vs. six and one half). j 4.1.1.3.2 Backfeed Through Unit Auxiliary Transformer
<
Because of the differential phase "B" fault lock out that occurred in the j startup transformer, the licensee consioered it prudent to perform ) electrical tests on the high and low voltage sides of the transforme In addition, an oil and gas sample was obtained from the transformer for i analysis. As shown on the 1:00 p.m. schedule of events in Appendix E, the restoration of power utilizing the startup transformer was initially I estimated to be accomplishable by 5:00 However, the recovery team
- _ _ _ - _ _ - _ _ - _ _ _ _ - _ _
r
]
)
19 i
!
determined that it would be advisible to proceed with efforts to use the backfeed option for recovery of power to the station buses in the unlikely event of damage to the startup transformer, i The backfeeding of the station is an abnormal operation that is performed when the unit is shutdown, and the startup transformer is to be taken out ; of servic The unit auxiliary transformer .is used to supply 4.16 kV loads. To accomplish this, the main generator iso phase bus quick dis- , connect links are removed, and the main and unit auxiliary transformers I are energized from the ring bus in the switchyard through ACB's No.104 and 105; thus, supplying power to the 4.16 kV system. However, the stuck breaker trip of ACB 104 earlier,in the day would allow only the use of ACB No. 10 Efforts to backfeed the 4.16 kV buses through the main and unit auxiliary transformers were controlled through Maintenance Request (MR) 87-73 This MR implemented maintenance procedure 3.M.3-9,. Rev. 3, Iso-Phase Bus Quick Disconnect Links Removal / Main and Unit Auxiliary Transformer Bcck- { scuttlin Both FSAR Section 8.2.1.2 and Technical Specifications Section l 3.9, Bases, make reference to the ability to backfeed power from the switchyard to the station auxiliary loads by isolating the generator from the main and unit auxiliary transformer The original backfeed plans envisioned power availability for the 4.16 kV bus to be completed by approximately 2:30 a.m. on November 13, 198 However, the following factors resulted in availability of this power source by 11:00 p.m. on November 12, 1987: 1) Staging was already in place under the iso phase bus duct; 2) The person directing the quick disconnect link removal (an NED engineer) was experienced in this task; and the related isolation of equipment required by the procedure; and 3) switchyard washing was completed in a timely manner. The tagging required ' by the procedure to allow disconnect links removal was accomplished at approximately 3:00 and the links open at 5:00 Even though washing of the switchyard would continue until 8:00 p.m. , the critical path would be completion of the procedural requirements in 3.M.3-9 that would align plant equipment to support the backfeed. The main transformer disconnect T930 was ready for closure at 10: 50 p.m. but, because of a linkage problem the disconnect could not be closed until 10:58 Additionally, the initial attempt to close ACB No.105 failed because a synchronization switch was not placed in the "0FF" positio At 11:09 p.m., ACB 105 was closed and backfeed power became avaliable to energize the 4.16 kV station auxiliary buse .1.1.3.3 Restoration of Power Once the backfeed of power to the main / unit auxiliary transformers was complete, the plant operators initiated re-energization of the 4.16 kV auxiliary buses and their associated 480V buses. The sequence of the auxiliary bus restoration was A6, A4, A3, A2, and A1. Auxiliary bus A5 continued to be supplied by the "A" diesel generator.
- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . ._ _
- _ _ _ _ _ _ - _ _ - - _ _ - . - - _ _ - - --- _ _ _ _ I
The foregoing discussions reflects BECo's restoration of power via the backfeed option. A parallel effort, as shown on the schedule of events in Appendix E, involved the return of the startup transformer to servic The NED recommended that testing should be performed on the startup trans-former prior to its re-energizatio Plant personnel had noted that the startup transformer miscellaneous alarm had not been activate This alarm provides monitoring of transformer parameters such as: oil level, oil flow, hot oil temperature, combustible limits, and sudden pressur . I The faut that this alarm was not actuated, combined with the knowledge from past events at the station that close-in faults had actuated the startup transformer differential protective relaying which resulted in lockout actuations, provided an indication that the transformer had not experienced damage. Because plant conditions were stable with extremely low fuel decay heat, the licensee directed that the transformer's secondary side windings be insulation resistance tested and the primary side have a Hi-Pot test conducted. Although this testing was expected to be completed by 5:00 p.m. on November 12, 1987, unexpected problems developed that ultimately delayed the availability of this power sourc The first problem was a determination that Hi-Pot testing could not be performed due to unavailability of equipment at the station. This situa-tion was rectified by the station's request for Doble testing equipment and personnel be brought to the plan The Doble test equipment was onsite and setup by approximately 6:30 p.m. Once the crew was onsite, it was determined that the lightning arrestors needed to be disconnecte These two conditions added significant delay in testing of the trans-fo rme There was no existing station procedure to control the Doble testing on the transforme All testing performed was controlled and documented under MR 87-731, TP 86-113, Rev. O, Meggering Procedure and 3.M.111, Rev. 1, Routine Maintenanc Once the startup transformer was satisfactorily tested, which was approximately 1:30 a.m. on November 13, 1987, plant personnel continued activities to energize the transformer and make it available as a power source for the station auxiliaries. By 3:35 a.m., the startup transformer was energized and available for service. Because the plant was in an off-normal lineup at the time, the licensee developed and performed an Opera-tions Review Committee review on November 14, 1987 for Temporary Procedure (TP) 87-252, Re O, Station Power Restoration from November 12, 1987 Inciden This TP provided guidance to the Operations Department to restore the plant to a configuration such that all station power was being supplied by the startup transforme The initial conditions were that buses A1, A2, A3, A4, and A6 were being supplied from the unit auxiliary transformer via backfeeding of the main transformer. Bus A5 was being powered from the
"A" diesel generator. To accomplish these tasks, the procedure reiterates the normal recovery from the unit auxiliary transformer as outlined in Procedure 3.M.3-9 for Bus Al-A4 and A6. For recovery of Bus A5 from the
"A" diesel generator, steps which were outlined in Station Procedure Number 2.4.16 were use ,
- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-_ _ _.._ .___ .__ _ . _ _ _ _ _ _
la
At 4:35 a.m. on November 14, 1987, the station power was normalized on the startup transformer and the backfeed was secured by opening ACB 104 and 105 in the switchyar I An estimate was made by the AIT as to what a reasonable recovery time would have been had plant conditions warranted an expedited restoration of offsite powe It appears that: 1) with the startup transformer differential lockout being perceived by the licensee as resulting from a condition that was not damaging to the transformer itself; and 2) offsite power reliability necessitated switchyard washing, it was concluded that between 1:00-2:00 p.m. on November 12, 1987 the startup transformer could have been returned to servic .1. Failure History Since the Pilgrim Station went into operation in June 1972, the station's 345 kV offsite system has experienced twenty events, including the November 12, 1987 event, which have caused loss of 345 kV offsite power to the station's onsite electric distribution syste Of these twenty ; events, five were caused by a single line outage occuring during the same time frame that equipment was out of service for maintenance or modifica-tion, five were caused by lighting, four by snow and wind, three by insulator salt contamination during ocean storms, and three by other miscellaneous cause In addition to the 345 kV offsite system, the Pilgrim Station maintains a secondary 23 kV offsite power line from a neighboring utility to the onsite safety related distribution system. This 23 kV offsite line is designed to automatically carry the stations safe shutdown loads given loss of the offsite 345 kV and onsite diesel generator systems. The unavailability of this 23 kV line and the 345 kV system will cause a total LOOP event at the statio The 23 kV line was unavailable at the Pilgrim Station during four of the twenty events described above. Three (3) unavailabilities were caused by snow and wind and the remaining unavailability (caused by its being tagged out of service for maintenance and modification during the November 12, 1987 snow and wind storm) may also have been caused by snow and wind. A total of four LOOP events have, thus, occurred over the past sixteen years at the Pilgrim Statio The frequency of total LOOP's to date at the Pilgrim Station, is 4/16 or 0.25 per reactor year (counting the November 12, 1987 event).
, 4.1.1.5 Changes and Improvements Since the Pilgrim Station went into operation in 1972, the licensee has initiated and completed a number of changes for the purpose of improving the overall reliability of the 345 kV offsite syste _ _ _ - _ _ _ _ _ _ _ _ . .
On March 15, 1979 the licensee initiated a design modification to the 345 kV transmission system in order to reduce the number of double line outages (with consequent reactor trip) being caused by lightnin This modification was completed on March 30, 1980. Prior to modification there had been three double line outages due to lightning. Follow modification, there have been no double line outages due to lightnin In 1980, the insulators on the Nos. 342 and 355 lines subject to salt contamination, were replaced with RG unit This type of insulator has a resistance glaze which produces a leakage current of about 0.5 milli-Amperes / unit when connected to an energized line. The design principle is that the leakage current will keep the insulator warm resulting in more uniform and rapid drying during fog or mist condition. Hence, the high voltage gradient across the insulator's surface, normally associated with the uneven drying on contaminated conventional insulators, is eliminated i and in turn the possibility of flashover is reduced. The licensee believes ' that this modification has significantly reduced transmission.line flashovers caused by surface contamination since 198 A program for routine inspection and periodic washing of the switchyard insulators was initiated to reduce flashover incidents caused by salt contamination buildup. The licensee found that this technique has been effective in reducing flashovers attributed to slow buildup of contamina-tion, but has not been as effective for a severe buildup over a short time. Thus, the licensee investigated the use of a new silcone elasto-meric coating for application on switchyard insulator Industry experience with this new coating has shown that it will give years of flashover protection on transmission lines even in locations with heavy industrial contamination, dust, pollution from highways, and salt fo The licensee completed installation of this coating on all but two switch-yard insulators in April 198 Operating experience, limited to the November 12, 1987 snow storm, gave a preliminary indication, based on an observed reduction of insulator noise, prior to the 2:06 a.m. loss of power, that the new coating will be effective in reducing flashover due to salt contaminatio Other changes which have been or will be completed before plant restart include modification of the startup transformer fast transfer logic, modification to protective relaying, and installation of monitoring equipment for better analysis of 345 kV system faults and breaker opera-tio .1. Conclusions and Recommendations Conclusions Changes implemented by the licensee for improvement of reliability of the 345 kV offsite system reduce, to the extent practical, the likelihood of 345 kV system failure due to weather condition _-___
_ _ _ _ _ _ _ _ _ _
The stuck breaker protection functioned as designed. The licensee is investigating the root cause of the indicated slow response of ACB 104 to , determine if the problem is with the main breaker mechanism, the auxiliary ) contacts, or the associated relays, and whether the problem is related to ! the cold weather. Corrective actions based on the findings of the investigation should improve the reliability of switchyard breaker operation.
Recommendations ! Misoperation or spurious operation of the differential relay for faults outside the zone of protection lead to lockout of the startup transformer requiring thorough investigation of the unit before returning it to service and loss of access to offsite power during that period. Hence, the root cause of the relay operation must be determined and corrected in order to increase the reliability of fast access to the offsite power source. The licensee is continuing the investigation of the relay opera-tion on the initial assumption of increased volts per-hertz being the root cause. However, the licensee has not investigated other possible reasons for the operation of the rela Current transformer performance, cali-bration checks and other operation and maintenance actions recommended by the relay manufacturer should be considere ; i Section 4.1.1.2 discusses the failure of plant operators to respond to the degraded 345 kV grid voltage alar Similarly, an observation in Section 4.1.2.4 of this report identifies an annunciated alarm that was not factored into the operator's analysis of the spurious primary containment isolation signal upon restart of the RHR pump These two events should i be reviewed and a determination made regarding the appropriateness of the operator's responses to these alarms and if a training deficiency exist ) Procedure 3.M.3-9 should be revised to reflect operational consideration for backfeeding with off normal electrical system lineups.
Develop procedures that describe and control testing activities on switchyard equipment and transformers.
4. Plant Electrical and Protection Systems 4.1. System Description - 4.16 kV Electrical System The auxiliary power distribution system (APDS) supplies the ac power required to safely shutdown the reactor and maintain it in the shutdown condition. The APDS also provides power to operate all auxiliaries necessary for safe startup, operation and shutdown of the statio The six 4.16 kV buses (Al through A6, see Figure 4) comprise both the emergency service and normal service portions of the APD The two emergency service buses, A5 and A6, supply power to essential loads _ _ _ _ __ _ . _ _ _ _ _ _ _ _ _ - _ -
- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ - _ _ _ _ _ - -
1 i required during operational transients and accident The four normal service buses, A1, A2, A3, and A4, supply power to other station auxiliaries requiring ac power during plant operatio The power _ supply to. the six 4.16 kV Vac auxiliary buses during normal operation is either from the unit source (unit auxiliary transformer X3) or the preferred source (startup transformer). Additionally, emergency
,
l buses A5 and A6 may also be supplied from the standby power source (diesel generators) or the secondary source (shutdown transformer, X13).
The eight 480 Vac buses, B1 through B8, are also divided into normal service buses and emergency service buse The essential 480 Vac auxiliaries required during operational transients and accidents are all supplied from the three emergency service buses (B1, B2, and B6). The l five normal services buses (B3, B4, B5, B7, and B8) supply power to other 480 Vac auxiliaries required during plant operation Secondary AC Power Source: The seconda ry ac power source provides an additional source of power to the emergency service portion of the auxilia ry power distribution system to permit portions of the 345 kV system to be removed from service for inspection, testing, and mainte-nanc It is connected to a 23 kV transmission line (No. 72) which is supplied from Commonwealth Edison Company's Manomet Substatio This substation is supplied by an approximately twelve mile long,115 kV line from the Horse Pond Switching Station. The incoming voltage is reduced from 23 kV to 4.16 kV by the shutdown transformer (X13). The shutdown transformer may be connected to the emergency service buses A5 or A Standby AC Power Source: The standby ac power system consists of two 2600 kW, ALCO emergency diesel generators (EDG's), each of which is capable of providing sufficient power to satisfy the loads on its respective bus (A5 or A6). Control power required for the startup and operation of each EDG is supplied from the 125 Vdc syste Other auxiliaries necessary to ensure continuous operation are supplied as required from the EDG through the emergency service buse Each EDG with its associated auxiliaries, connection to 4.16 kV emergency switchgear, control system, and distribution of power to the various safeguard loads, is segregated and sepa rated from the corresponding systems of the other ED Each EDG is operated independently of the othe See Section 4. .1.2.2 System Description - Reactor Protection System and Primary Containment Isolation System ,
{
Two motor generator (MG) sets normally supply ac power to the RPS (see ) Figure 5). RPS MG set A is supplied ac power from 480 V, MCC B-23 which is l connected to 480 V bus B3. The RPS MG sets A and B supply power to 120 V RPS buses A and B, respectively. Alternate power to either RPS buses is from 480 V MCC B-10 (through transformer X-20) off 480 V bus B6. Controls and interlocks are provided such that only one RPS bus can be supplied by _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ -
.
480V MCC W-23 480V MCC B-22 _ 480V MCC s-10
.
..... .S .,-,,,.S
,
v (4
"
h (th S.-,... . RPS .
[" h RPS MG SET A
,0.TSKVA I M) . V I
(U) I
-
,
MG SET
<Ai dSO/,89V c.y' 1 sA> 18.T S K V J
,8;Y coH
~S m ,, (.y, _
..;Y VR ' "*
VR I I VOLTAGE VOLTAGE O O ~ ~ FREQUENCY q , FREQUENCY ,
*
PROTE CTION PROTE CTION r: 1 I I j PwR SUPPLY PwR SUPPLY PwR SUPPLY f PROTECTON PROTECTION PROTECTON , 1 ASSEWSLY ASSEMSLY ASSEMBLY SEISMIC ,h~~.) EPA 1 __ J EPAf ._ j EPA 3
]
UPGRADE , ASSEMsLIES l I I i
'
Pwm SUPPLY PwR SUPPLY PWR SUPPLY PROTECTCN PROTECTON PROTECTCH k ASSEMBLY ASSEMBLY ASSEMBLY
...) EPA 2 .. JEPA6 '
. J EPA 4
. . . - . . - - - . . . - . . . - - . . .. . . . - . . .. <
MSnuttiy OperSted
. . * *
Cs1A Cs2A Cgra
~ ~ ~ ~ ~ ~ ~
paie
~ ~ MbH ANSCAL #dERLOCK "" ~~ ~ 3~ : :
L . . .--. . TR ANSFER 8 WITCH C-511
,
l
. .- -. . . . . . - - - . . . - - - . . . - . . - - - . .-...J 0,, @ @
fs
,, . ,,
RPS SUS 'A' _ RPS SUS 'S' OII E e-
. CB1A CB2A C828 CB18 C-511 TRIP LEVERS :@ Determine. which Suppil . RPS 'A'
hDet.tmine. which Bu. Rec.lv.. Stensby Power
@ D.i., min. . .n..h S. .n. . RP s s -
120V RPS POWER SUPPLY SYSTEM
FIGURE N l
*
_ __ _ _ __ __ - _ - _ _ _ - - _ _ - _ - - _ _ _ _ _ _
the alternate power supply and only when normal MG set power supply is not connecte Each power supply to the RPS buses is protected by two electrical protection assemblies (EPA) in series. These EPA's provide overvoltage, undervoltage, and underfrequency protection to the buses and connected equipmen Each 120 V RPS bus supplies power to the associated RPS Trip System, Power Range Neutron Monitoring System, and Offgas and Steam Line Radiation Monitor The A-RPS Trip System bus (Panel 915) supplies the power requirements of the A-RPS Analog Trip Cabinets (C2228-Al and C2228-A2), Channels Al and A2 Sensors, Channels A1, A2, and A3 Scram, System A Scram Reset, System A Pilot Scram Solenoid Valves, and PCIS Trip Logic Channels Al and A2.
, Similarly the B-RPS Trip System bus (Panel 917) supplies System B power i requirements, including B-RPS Analog Trip Cabinets (C2229-81 and C2229-82) and PCIS Trip Logic Channels B1 and B2.
l The PCIS, Group III isolation will isolate the shutdown cooling mode of the RHR system for any one of the following conditions:
(1) Reactor low water level, (2) High reactor vessel pressure, and (3) High drywell pressur There are two 120 Vac RHR Isolation Control circuits, one for channel A and the other for channel B, for isolating the RHR suction valves 1001-47 (outboard, 125 Vdc valve) and 1001-50 (inboard, 480 Vac valve off MCC B20). Channel A is powered from safeguard 120 Vac panel Y-3 off 480 V MCC B17, and channel B from panel Y-4 off MCC B18. (Figure 6). These panels are supplied power from the diesel generator buse As shown in Figure 6, when relay K29 deenergizes valve 50 will get a close signal. Similarly, when relay K30 deenergizes, valve 47 will get a close signal. Relays K29 and K30 deenergize when relays 5A, 5B, SC, and SD deenergize in a one-out-of-two-twice logic. These relays, which are in the PCIS Trip Logic Channels, will deenergize when relays in the RPS sensor channels sense low reactor water level or high drywell pressur Relays K29 and K30 will also deenergize through relays K28 and K50. These K28 and K50 relays are deenergized on high reactor pressure as sensed by pressure switches 261-23A and 261-23B, which open on high reactor pressure. If either relay K28 or K50 deenergizes, both 47 and 50 valves will clos All these relays will also deenergize on loss of power of the RPS, Y-3, or Y-4 and will close the dc valve 47 and ac valve 50 (once power is returned to MCC B20 via 86).
- - _ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _ . _ _ - _ _ _ . _ _ . - - . - _
1
, .,
.s
' . . . ,
2e 5s
> > "I I ,
w .
.... a a
m o o o O O o
'**$ x n g M e ,
.
De
D" m 9 '
~o 2 n* 2e za8 I '
i ewe
*"* > O s ?.
C
,
.
.S S <
e C * .g . CL
* b '. * eo eO
__, c a 2 .-. wom O .8 O* eo O* e 3aH 3 e
>
mm N Oe g e M 9e , OC o o . E g >
-U 2 U 'o ( o r* - E. -o e
-
O$r # ( Q ( L
=xa e e
~ m
. . .F n. m OOw D
n E
* u e
;
= E
*: *: 6 I e
O :
,
-
, a es a
-
. - ai -
-
. :
\- =
-4 H HHF
.. s
.
p F- {, Y gC
. . . . :: e : , o
_ y I *HHF?. + $
:
, ,
<
e, i * a e - .
.
* *
,
,
-4: F--4 Fi H Fi F4 -4: 'F HF u : = .e
.b *
i
.
dHF
. .
^v-
.
e e M E E E l <
*
.
g ' mjs
.
o8 y. h o R .
*
i **:: = = l
:less:Y
~
.S
. - . i i
.
0% ke>. NO 0%
.$ 4 NO D
,
e* l
---__-_____m_ _ _ _ .
.;. % t l x
>>
,
27 \
(
\ '
To reset"the isolation, the initiating isolatMn signals must & cleare ~ Power must be available to at least one of thd two RPS buses and to both Y-3 and Y-4. The reset switch for the 'RHR shutdown cooling isolation valves (switch 16A-533 on panel 905) must then be operated to energize reset relays 16A-K11 and K12. (To enyrgize relays K11 and K12, power to panel Y-1 from either MCC B10 or BIS muy.t be available.)
4.1.2.3 Operations and Failures During Event ^
','
\t8 t Y Initially all 4.16 kV buses vre energized from the startup source. All the unit source breakersies open. Bot.h diesel generators were operable ,
!
O '
.
and in standb The shutdown transformer was not available due to maintenance which included tie-in of the new "b'.ackout" diesel generato / All 480 volt buses and mEor control centers (MCC's) were energized with i the exception of bus B26 for the intake structure. Bus B4 was abnormmaly aligned and supplied from the 62-B4 cross-tie instead of the normal feed from bus A Bus B6 was being supplied from bus B When the loss of power occurret both of the running Reactor Protection System-Motor Generator (RPS-MG)' sets were de-energized causing a full scpn sMnal (channels A1, A'E, B1 'nd B2). . In addition, a Primary Cont intent Isolation Signal @F) was generated due to the above channel triph is well as the temporag interruption of power to buses, Y3 and Y The computer shows these trips coming in just before 2:07 a.m. on November 1 The PCIS caused residual heat . remq' val (RHR) motor operated pump suction valves 47 and 50 to close and t% operating "A" RHR pump to trip. Bus Y3 and Y4 were then re energized as both EDGs energized their associated buses. An operator was dispatched to energize the "A" RPS bus from bus ; B10. At about 2:21 a.m. , the channel "A" trips reset and this allowed l reset of PCIS. Re-energizing one full RPS channel clears the "one out of two, twice" logic. At about 2 30 a.m. the operators atten'pted to restore shutdown cooling. When "A"i lHR pump was started, another PCIS occurre The operators did not notice the 2:30:59 comput- }og alarms which ! indicated trips of several channel A2 parameters. s . lese trips were the first indicator of the fuses which blew in the anal Q trip system cabine In all, four fuses blew, two on channel A2 and one a each channel Al and Bl. The A2 fuses are located in cabinet f222A-A Only the A2 fuse failures caused a plant response since each cabinet has dual power supply feeds via separate fuses off the same circui An operator was sent to the MG set room where he reported that "B" MG set j l was running on its normal supply but at lowered voltage and curren The '
"B" MG set would have been disconnected from the "B" RPS bus by the electrical protection assembly (EPA) and disconnected from its power supply when power was lost to bus B22. An interriew with the NOS indicated , that although ne recognized that the load on "B" EDG was only about 1300 q kW, half of rated load, he felt that the diesel was overloaded causing '
reduced voltage and current on the MG set. In resoonse to this perceived condition, the NOS dispatched another operator to strip bus B4 and open
, .,
_ . _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ . _
~'
.p q <g4 .
-
a
\ y l
/,
f
! ,;
1' :7p "f ! ypl [ 28 . i
,
j
.the feed to . 50 from ' bus se. . Bus B4 1. . e non-safety bus not normally I powered from SObut cross-t%I because W? normal A4/B4 feed was out of J service. 5 hwy 1 thereafter,- the NOS ordered "B" RPS bus transferred to ;
j
.l B10, de energizing the "A" RPS bus a ni.' causing another PCI This !
occurred e.t abcut 2:37 a.m. on November 12. ; Once again, a review of the
'
computer alarm logs clearly shows that only the Al trips came in when RPS PA" wu de-engized; the, A2 trips remained in due to the un-noticed blown - l fuse d With RPS "B" tik . powered from BIO, the channel "B" trips reset, 41
'PCIS was' reset and "A'? and "C" RHR pumps restarted to establish shutdown I '
cooling. In spite of -the confu<icn over thf RPS behavior, the operators were able to re-establish shutdown cooling by' 2:45 a.m. , about 40 minutes after the trip which under worst case decay heat conditions, would have ., had ne safety effec '
,
,
a I # It should be noted thn, an long as buses Y3 and Y4 are energized (i.e.
h both diesels running); fit only takes one' half of. the RPS .to reset PCIS.
J,
,, The ' blown fuses werr ~ not found until 5:55 a.m. on Novenbar 12. This TN ' *
scena 'comes moreJ significant if the blown fuses occur simultaneously _ with a mesel f ailure on the opposite trai In this instarce with no O > power to Y4, the shutdown cooling isolation valves would have to be-1 electrically jurse ed to open the The "B" diesel was available 1 until 11:30 a.m, that mornin .s?. ,.2J * Conclusicais mWRecommendations ( , ,c There was confusion in the control room concerning the loilding on the "B"- diesel genentor and on what actions should be taken to restore power to
:he'RPS bus s The operators did not notice the fuses blowing on channel A even though they were annunciated on th4. comp' uter. This failure to-notice or, acknowledge computer alarms was also evidenced in the 345 kV system at 1:27 A.M. on November 12 when the line 342 low voltage alarm cams iri on the compute It appears that operators are not routinely analyzing; computer alarm data ta ~ assist in assessing plant status. 'A revfew of the operator actions. should be muie to assess the appropriateness of operator actMns and their analysis of plant data for-this event, y_ *
l r- The 10 ano fuses that blew in the Analog Trfp Cabinets - one in C2228-A1, two -in C2228-A2 and one in C2229-B1 - were apparently due to a common cause connected with either the starting of the A-RHR pump or the ( _ switching of the RPS power supplies. The licensee is investigating the roct- causes of the blown fuses. Had the other fuse in C2229-B1 also
' .
blown, operators would not have been able te reset kHR shutdown cooling isolatio Corrective. actions based or the results of the investigation should be initiatec; and completed before reactor restar During the event, when the B Diesel Generator was out of service and panel
'
( Y-4 was without pow r, the licensee initiated a temporary modification t scheme to provide poser to the control circuit of.the RHR suction outboard
.
.
I h
?
c , fy' ! ', f
" '
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -
.
.
. _ _ _ _ _ _ .
, _ _ _
29 l i I1 valve 47. Contingency procedures for the single failure of either onsite
-
emergency power system train (i.e. , loss of either Y-3 or Y-4), should be l considere In reviewing procedure 2.4.25, Loss of Shutdown Cooling it was noted that the immediate operator actions provided no specific guidance on action required to restore shutdown coolin Similarly, the subsequent operator actions did not specify any mitigating actions to be taken for conditions other than full buses A5 and A6 and PCIS logics availabl The adequacy of this procedure should be reviewed relative to the LOOP even .2 Emergency Diesel Generators 4. System Description Each emergency diesel generator (EDG) unit is an 18 cylinder, 3485 hp, 900 rpm engine powering a 2600 kW, 4.16 kV, 400 amp, 60 hz, 3 phase generato They are located in separate rooms of the diesel generator building which is north of the reactor building. They may be operated at either the local control station or remotely from the main control room. In a loss of power situation, they start automatically and energize their associated buses, A5 and A6, within 10 seconds of the time the following conditions > are met:
-
startup transformer supply breaker open, startup transformer secondary voltage is low and unit auxiliary transformer breaker racked out, or
-
unit auxiliary transformer breaker is open and the startup i transformer breaker is racked ou <
)
The EDG will automatically start, but not assume loads unless electric ] power is lost, upon receipt of low-low reactor water level or high drywell pressure signa The EDG lube oil system is provided with a 480 Vac pre-lubrication (pre-lube) pump which circulates lube oil through a 480 Vac electric heater and then though the normal lube oil flow path. This keeps the engine warm and properly lubricated in order to be ready for an automatic star Both the prelube pump and lube oil heater re supplied from emergency 480 Vac motor control centers, B17 and B18 for "A" and "B" EDGs, respectivel Heater failure is annunciated locally on the control panel when oil temperature drops below 70 F. Pump failure is similarly annunciated when pressure drops below 2 psig. On an engine start, the prelube pump stops automatically when the engine driven jacket water pump raises pressure above 10 psig. The prelube pump will automatically start when pressure drops below 10 psi The heater is interlocked with the prelube pump to prevent overheating if prelube forced circulation is not available. The I
;
'
_ _ - _ - _ _ _ J
heater temperature is controlled by a thermostat switch located on the heater assembl Lube oil temperature and pressure gages are located locally on the control panels, one panel per engin ) Instrumentation for diesel generator output current is provided through a current transformer. The current transformer uses a 600/5 turns ratio; , this circuit is shown in Figure No. 7, and is comprised of an overcurrent I relay and ammeter for each of the output phases of the diesel generato '.The current 'from phases A and C are used in a kilowatt hour (KWH) mete All circuit components, except for KWH meter, are located in the 4.16 kV switchgear compartment associated with the respective diesel generato The KWH meter readings 'are recorded as part of surveillance testing activitie .2.2 Operations and Failures During the Event 4.2. Current Transformer l At approximately 10:00 a.m. on November 12, 1987, an I&C Supervisor in the-vicinity of the "B" EDG output breaker compartment in the A6 Bus switch-gear room noted a zero current. reading on the phase C ammeter. This condition was reported to the control room, and ' the operations department issued MR 87-730 to investigate the observation. By 10:45 a.m., the licensee's Electric Lab Engineers were troubleshooting the circui From amp probe measurements taken at the A609 breaker compartment in the lower switchgear room, they determined that current was neither flowing through the ammeter nor the overcurrent trip relay of the phase C ci rc ui t . Verification was made using other switchgear and 480V load center ammeters that the EDG was not operating with unbalanced loading on its phase Subsequent to the field troubleshooting, the Electric Lab Engineers observed the operation of the KWH meter for both EDGs, which are located on Control Room panel C-8. It was observed that the "B" EDG KWH meter was running at half the speed of the "A" EDG, yet both EDGs were evenly loaded. At this point they concluded that the phase C current transformer circuit for the EDG was either open or short circuite The trouble-shooting and possible failure modes were reviewed with the cognizant NED Engineer during a call from the control room by the Electric Lab Engineers. The NED engineer concurred with the findings and supported the Electric Lab Engineers recommendation for removing the EDG from service and isolating the failur Because the possibility existed that the current transformer was operating with its secondary side open circuited, which is a potentially dangerous condition that involves the development of a very high voltage in the secondary circuit, the Electric Lab Engineers informed the NWE of their recommendation for securing the "B" EDG at 11:20 a.m. The EDG was secured from service at 11:35 a.m. and isolation of the unit completed at 1:30 to support additional troubleshootin Further investigation determined that a wire was disconnected from its terminal lug at the phase C KWH meter output terminal (No. 8), which is shown in Figure No. 7. The l _-__-____-_--_-_--_--_D
- - _ _ - _ - _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _
.
A.l4 kV Bus as G&A 47HGn'7" l ' ,...
. -
IH l
"
<
l jw/s . C -
, u , o G enenp ron eu w r V)
' 84WA WG C 1rtr9r
#4 i *
I )tp C o #A g N i~ j %g-- & lt< a , 7*4'nds feen c it s
- h it 9 .
CeNTel- Reap J% *
-+ - .
" N,,
PA NE t- C8 , '&.
.
b &
-Ol* SE L L e.
vE " . o t * F ._g -= .= ' RR 2 e EE 1 TV "- 1 9 F t - A iF X "e - O t n eE nY eR R - _ I gl- 'tIg i O _ st uN s A n5 ; _ , R E Y ,3 E0 Y8 - - x',gO n r R O , , r R-OK t 0 A U F . N R E F t T sRiO EAL R S - l C - I 2 a s t , R T "- 1 ,. a - a ct $ 1 o mc Qr- . % ' C E L E " , j M - A5 a : ns e a a eo 1 R g t n a 7$0 > E I ' i k ~ ' + i\ t i ,P + R < ^^ h r ,E v , ( ,Y J / ,R ,D ,1 I A l j ,M , 8 r ,A ,l " -- D . E a ~ S l f X M ~~- e 3 S E E R r n0 H t n q R U G m ',y @s, *, u P I F C, o v4 I 9 8 -- p,3 f - k .g3 C M : I pi 1 e"r" R -- O 4 3 ,l- . e 8 ee t 2 1 gftE -.r . C ? s , ] t s X - pt F . . ' ' .. -; - - d ;-, . X gc . ' , . .- .- ._ 7- y WW Wy gs PO ' k 1 ) .- R . '*. 7 n' y V C g - C I - # E - C R G I - 8 A o - " 4 4 0 0 t g d> 1 1 . - - g C ( ) Y MM p e, , V AV t t , { y '. c c , t 8 e e e, - m o ot . t m , .- g R E o g - ,j - - a n , , N . T e e . e- r ( s s . $, ' g A e e . , .v ' I l = e ae g g - - H s t S( n , a O S S. t T,d , , 1 j - R SO n I O R R Et t r t i ' ii: l ,R o E R S I rN , t E C s E , , A S uO 4 s T. C E _ OC s a xC E N 1 E n , t s E N , 4 , e n ij" C , f g A r e , E - r e t uuo C oje g t M s C s i u c P 'au o , l C T mn[s , w C e? l s __-_ Component Description There are two main screw-type compressors (K-110 and K-111). One main compressor can normally supply the entire station load. The compressors are cooled by turbine building closed cooling water (TBCCW). Air com-pressors K-110 and K-111 are powered from 480 V load centers B-8 and B-5, respectively. The maximum working pressure for K-110 and 111 is 152 psig and the normal working pressure is 102 psig. At 102 psig the compressor capacity is 655 cfm. The screw compressors have an internal outlet check valve and a check valve in the discharge line leading to the receiver tank inle The check valves prevent system pressure loss by compressor leakag There are three smaller air compressors (K-104A, B, C) which are vertical, single-stage, double-acting reciprocating compressors, rated to deliver 160 scfm at 105 psig (see Figure 9). Air compressors K-104A, K-1048, and K-104C receive power from 480 V, MCCs B-15, B-14, and B-10, respectivel A normally isolated pressure reducing valve in the cross-over line, PCV 4370, is installed between the high pressure distribution header (up-stream of the air dryer) and the low pressure distribution header. The cross-over can supply low pressure service air if the low pressure blowers fai Some systems which use instrument air as the motive force for a com-ponent's operation, have a check valve and an accumulator in their suppl This ensures that the components will continue to operate even if HP air header pressure is lost. Seismic class I accumulators, associated piping, and check valves are provided for the following equipment: * Torus to secondary containment vacuum breaker butterfly valve * Main steam isolation valves * Main steam relief valves Service air header valve A0-4353 isolates the high pressure service air to low pressure service air header cross connect at 85 psig (decreasing) if a problem occurs in the high pressure air system. This ensures that vital services in the high pressure air header will receive all available ai Valve A0-4350 isolates the high pressure service air header, in case of a leak, when the upsteam pressure switch senses 85 psig (decreasing). Valve A0-4365 is shut to isolate the non-essential instrument air header, in case of a leak, when the upsteam pressure switch senses 80 psig decreas-ing. The detailed system interrelationships and effects of losses of the various high pressure service and instrument air headers are presented in Appendix .3.2 Operations and Failures During Event Operating Procedures for Air Systems Instructions for operating personnel to perform normal operation of the instrument and service air systems are provided respectively, in PNPS _________________a l i Il ]l ' ^- O' A $g _E * ? t t e e f R m t y E l v f/ y- N t - ' E E C C -l E E R R _ . g N N g- 2}v j N . N . O a t / E < E V L A V Y O R - , R-v t E T T E T A ,E t t= C N N ~ F ,V' E R M E A L 8 A P ,A",.. ,V . E . E M S " "- - - A i R E E G . S W C f f A E R 6= . . M N A R g E Mf Of A t M R A E N T T N U S h O C ' A = :=E" A @' E E G N I P I . R O M A R T' ' E P O N E t A S S 't d W" R E R E E O U R P M O "c n E / S S G I F E C E l f en o r - _ dia, / ' E R P Y A W n C i , ( E M O A E P h T.- S S E = # C p/ - O n L ] S L '# e g E A R E T L '- ;- ' .M C \ w Ae R @@Kc - l N E C I P Y T FI D O n e r- O M # O M R # E H T R E , , t , ,E , t E E A f , t E t R , u i , O # W F e R e (=A I A D O O g # H N x Atg \%' hsQKh7 # e , , , - U - - t g8 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - _ _ _ - _ _ _ _ _ _ _ _ _ _ _ Nuclear Operations Department (N00) procedures No. 2.2.36, " Instrument Air System, " Revision 15, dated October 23, 1987 and No. 2.3.37, " Service Air System," Revision 9, dated July 3, 1987. Those procedures appear to provide an appropriate level of detailed and clearly written instructions to assure that operating personnel conduct proper system lineup and component operations for an operating plant condition. No. instructions, however are included in those procedures for abnormal operation of the air systems. Operator actions for emergency operation of the instrument and service. air systems are provided in PNPS NOD procedure No. 5.3.8, " Loss of Instrument Air", Revision 10, dated October 30, 198 Procedure No. 2.2.36, " Instrument Ai r Systems", is written for a normal operating plant condition during which all five air compressors would be expected to be operabl [As noted in the procedure, however, there are no technical specification limiting conditions for operation or admini-strative limits regarding instrument air system components.] One main air compressor, K-110 or K-111, is capable of normally carrying the station's entire load while -in the leading position control. The other screw com-pressor will be in the lagging position control. The leading compressor cuts in at 102 psig (decreasing) and cuts out at 108 psig (increasing), whereas the lagging compressor cuts in at 93 psig and cuts out at 103 - psig. Two of the smaller units, K-104A, B or C, are normally in the " standby" position control and will either cycle one at a time or together dependent on system pressure deman The other smaller compressor is normally in the "off" positio Air System Configuration and Status on November 12, 1987 On November 12, 1987, prior to the loss of oftsite power (LOOP), the instrument air systems were not in a normal operability configuration, as shown belo Instrument Air Systems Operability Status Prior to LOOP Normal Power Emergency Power Operability Component Source Source Status ] Compressors K-110 B-8 None On - Lagging, tagged for known oil leak K-111 B-5 None On - Leading K-104A B-15 "A" EDG Tagged out of service for compressor i replacemen ] l _ _ _ . . _ . . _ _ . . _ _ . _ . _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ . _ _ _ _____J - - _ - _ _ _ - _ _ _ _ . K-104B B-14 "B" EDG Control switch-off not tagged, but known head gasket air leakag K-104C B-10 "A" EDG Tagged out of service for: (1) TBCCW (aftercooler) air leak (2) electrical ground (3) oil leak (4) failed capacity tes Air Dryers X-105 B-10 "A" EDG On X-160 B-23 None On - not cycling properl Losses and Restorations of Instrument Air System When the LOOP occurred at 2:06 a.m. on November 12, 1987, on-site power was lost to the screw compressors K-110 and K-111. Although both the "A" and "B" emergency diesel generators started as a result of the LOOP, none of the reciprocating compressors started as two were tagged out of service and the control switch for the third was turned of Thus, with no operating compressors, a total loss of instrument air occurre The resultant reduction in air header pressure caused block valve isolations to occur within five to ten minutes after the LOOP. Air header pressure continued to drop off to about zero psig in the next fifteen to twenty minute At 3:10 a.m. compressor K-104B was started at the local control panel, and the essential instrument air header slowly repressurized to 40-50 psi The air system remained in that conditien until 11:30 when the "B" EDG was secure That action caused compressor K-104B to stop upon loss of its emergency power source, and air system pressure again depressurized ; to zero psig in 20-30 minute The air system remained depressurized ' until af ter of f-site power was restored and . screw compressor K-111 was i re-energized at 11:45 The instrument air system achieved a normal l 102-108 psig operating pressure band at 12:05 a.m. on November 13, 198 Except as a result of air compressor inoperability at the time of the loss of off-site power or loss of the normal and emergency power sources to the l air compressors described above, no other failures occurred to instrument air system components during the total or partial losses of the compressor All system header isolations and system interactions occurred as expected for the cold shutdown condition of the plan _ -__ ______-___ _ _-_-___-_____--_-_ - __-_ - _ _ _ _ _ . _ _ _ _ _ _ _ _ _ ._ - _ _- _ _ _ . _ _ _ _ _ _ _ 4.3.3 - Air System Recovery Operator Response Procedure No. 5.3.8, " Loss of Instrument Air," is written for the expected plant response to a degraded or loss of instrument air header pressure that occurs during a normal operating plant conditio The procedure identifies automatic and operator actions with respect to feedwater, condensate, control rod and cooling water system operation. The procedure specifies that subsequent operator action should include (1) the initiation of efforts to re-establish normal instrument air pressure, and (2) the isolation of makeup to cooling water systems surge tanks. Both o . those actions were taken by plant operators af ter the loss of off-site power occurre Within one hour of the event initiation, operators verified that proper air system isolations had occurred, manually isolated makeup to the cooling water surge tanks and started compressor K-104B, the only available air compressor. When air header pressure was restored to 40-50 l psig, plant operators checked that air system component configurations ; were unchange For the remainder of that shift, plant operators maintained the instrument air system in a degraded, single-compressor, i mode of operation, while periodically checking the status of air system ! conditions during plant tour The air system status was also confirmed by the relieving day-shift operations personnel soon after turnover of the shift watch station Emergency Procedure Adequacy / Procedure No. 5.3.8, " Loss of Instrument Air," contains the essential operator actions to place the plant in a stable condition, in the event that the instrument air system becomes degraded or inoperable while the plant is operatin However, both licensee and AIT personnel noted, during the review of the LOOP event, that the procedure does not address subsequent operator actions with specificit For example, the procedure does not address actions to be taken when the essential air system header remains in a partially pressurized, as compared with a fully depressurized, conditio Also, the procedure does not provide instructions for operators to verify specific component response functions have occurred upon loss of instrument ai Licensee management has directed that procedure No. 5.3.8 be reviewed for appropriate changes, as , a result of the lessons learned from the LOOP even ' Air System Restoration Actions At 7:30 a.m. the nuclear operations department manager directed the systems group to pursue options for restoring air system pressure, and a parallel path recovery plan was initiate An unplanned priority "A" maintenance request was promptly written to investigate and repair a suspected ground to the motor of compressor K-104 At the same time, - _ _ _ _ _ . _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - __, - _ - _ _ - - _ _ _ _ - . _ _ _ - . _ _ -_ _ _____ ____-_ _ _ _-______-_________-_ -__ __ - __ systems engineers determined that there were no suitable portable air compressors onsite that could be connected temporarily to the instrument air system, and a vendor (Atlas-Copco) was contacted to locate an oil-free compressor. Meanwhile, systems engineers developed a temporary modifica-tion for installation and use of an offsite air compressor. These efforts continued throughout the da A portable compressor was obtained, installed and tested by 10:30 p.m. , but was kept in a standby mode pending the near-term restoration of off-site powe After power was returned to in plant buses, compressor K-111 was re-energized at 11:45 p.m., and the instrument air header attained normal operating pressure shortly thereafter, as discussed in Section 4. Subsequently, after completion of the corrective maintenance for compressor K-104C, it was satisfactorily tested at 1:30 a.m. on November 13, 198 The recovery actions taken for restoration of the instrument air system are summarized in Appendix G and discussed further in the sections that follo Plant Operation with Degraded Instrument Air System , As indicated in the sequence of events, between 9:30 and 10:00 on November 12, 1987 the senior system specialist (system engineer) for air systems completed an initial walkdown of the instrument air system to determine proper isolation header and component response functions. Based on his knowledge of system design features, the engineer recommended to on-shift operations supervision that compressor K-1048 be stopped in order to fully depressurize the essential instrument air Leader. The basis for that recommendation was that it would be better to have the air system in a condition that had been previously evaluated as to the system response and consequences, as opposed to maintaining the header partially pres-surized at 40-50 psi Operations section management was still consider-ing this recommendation and investigating alternate methods of restoring the instrument air system wt en compressor K-104B was stopped as a result of manual shutdown of the "B" emergency diesel generato The appropriateness of operating the instrument air system in a degraded, partially pressurized condition or in a fully depressurized condition is being assessed by the licensee in conjunction with the review of procedure No. 5.3.8, " Loss of Instrument Air," as discussed abov Compressor K-104C Maintenance Activities Based on their review of outstanding maintenance requests and tags for reciprocating air compressor K-104C, operations and maintenance personnel attributed the electrical problem to a motor groun The priority "A" < maintenance request (MR) No. 87-827 was originated to investigate and I repair the suspected ground, using PNPS nuclear operations department procedure No. 3.M.3-8, "In;pection/ Troubleshooting - Electrical Circuits". The licensee's initial investigation determined that the fault was ) actually within the compressor's circuit breaker in motor control center j (MCC) No. B-10,. compartment No. B1064, and further work was conducted I _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ ' ' - - ' - - - - - - - _ _ _ _ _ . _ _ _ _ ! ) 43 J using procedure No. 8.Q.3-3, "480 Vac Motor Control Center Testing and Maintenance." The electrical fault probable cause was then identified as the A phase motor terminal wire that had lost contact, due to only a partial. number of strands that were landed in the compression terminal lug which in turn ] caused the wire to overheat and burn awa The wires were restripped and ' terminated on a new terminal bloc As noted on the procedure No. 8.Q.3-3, Attachment B, "480 Vac Motor Control Center Inspection and Maintenance," completed November 12, 1987, breaker B1064 had been reworkr3 during refueling outage No. 7. The A phase field wire to the ! motor apparently did not have enough strands wired in the compression ; terminal. The wire then burned frec and during the process heat destroyed the terminal . Therefore, the terminal board was replaced. No degradation was noted to other components, except slight discoloration of the field wire. While conducting the MCC maintenance post-work testing, compressor K-104C starting and running currents were obtained, but the compressor was not loaded because an oil leak was discovered on the lower crank case side cover. Priority "A" maintenance request No. 87-733 was then initiated by the nuclear watch engineer to change the oil pump gasket, check the oil pump relief valve and change the oil filter using procedure No. 3.M.1-1 During subsequent repair activities maintenance and system ' engineering personnel determined that the oil leak emanated from the oil pump itself, which was also replaced. In addition, a new cover gasket was fabricated. Post-maintenance testing was completed satisfactorily late on November 12, 198 Compressor K-104C loaded and unloaded within the design criteria, and there were no oil leaks at 30 psig oil pressur The post-maintenance testing was completed satisfactorily at 1:30 a.m. on November 13, 1987, and compressor K-104C was declared returned to normal at 10:40 a.m., after all tags /isolatiors were removed and MR logs and files were updated. The maintenance activities related to compressor K-104C were conducted methodically and properly, in accordance with approved maintenance procedure There was close coordination between maintenance and operations personnel of the various repair and post-maintenance testing activities, as well as continued oversight by the senior system specialist (system engineer) of the comp re <., so r MCC breaker and oil pump repairs. Subsequent to completion of the compressor maintenance, on November 16, 1987 maintenance group engineering initiated PNPS failure and malfunction (F&M) report No. 87-641 regarding the electrical failure of breaker B1064 as described abov The major concern identified on F&M No. 87-641 was that compressor K-104C was not operable when required. Further, that deficiency was identified on August 3, 1987, by nuclear watch engineer tag No. 46-227. The licensee's F&M No. 87-641 review, including determination of a requirement for root cause analysis and corrective action plan, was not yet completed at the end of the AIT inspectio The licensee's further actions regarding F&M No. 87-641 will be reviewed during a ; subsequent NRC resident inspectio ! ! _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ , _ _ _ ______ _ _ _ _ _ - _ _ Off-site Air Compressor Installation About 8:00 a.m. on November 12, 1987 station and engineering management - determined that there was no oil-free portable air compressor onsite that would , be acceptable for temporary installation into the instrument air syste Therefore, at 8:15 a compressor vendor, Atlas-Copco, was contacted to locate a suitable oil-free compressor that could be promptly . delivered to PNPS. The vendor's location and delivery of an off-site air compressor is summarized in Appendix ' Temporary modification- (TM) No. 87-30, " Temporary Instrument and Service i Air Supply", was initiated to provide a properly approved installation of the portable back-up supply of compressed air to the high pressure service air and instrument air systems to be used if both the K-110 and K-111 compressors are unavailabl Specifically, TM No. 87-30 provided for installation of a temporary high pressure compressed air supply to the outlet piping of compressor K-104 A. The TM No 87-30 was similar to TM No. 87-21, which had been approved for use during the recent months of the plant shutdown, while compressors K-110 and K-111 were unavailable for scheduled maintenance. The TM 87-30 was properly approved both by the onsite review committee . (ORC) during ORC Meeting No. 87-133 and by the nuclear operations department manager on November 12, 198 The nuclear watch engineer subsequently authorized installation of the temporary modification, using MR No.87-736. The TM No. 87-30 installation was completed as authorized and the back-up Atlas-Copco air compressor was operationally tested on the evening of November 12, 198 However, pending imminent restoration of off-site power and the availability of compressor K-111, the backup compressor was maintained in a standby mode and was not used to pressurize the instrument air syste .3.4 Changes and Improvements Recent organizational changes and an air system status review completed in October 1987 contributed to the licensee's ability to promptly implement sound recovery actions in response to the various losses of the instrument air system during the loss of offsite power even In early 1987, the systems group was formed as part of the technical section within the nuclear operations department of the PNPS restart organizatio As discussed in the PNPS Restart Plan, Volume 1, transmitted by BECo letter to the NRC dated July 30, 1987, the establishment of the systems group was patterned on INPO Good Practice TS-413, "Use of Systems Engineers". The systems group is to provide greater depth and continuity in the oversight of major plant operating systems at PNPS by assigning " ownership" of each system to a qualified individual. These personnel are also assisting in the restart effort by reviewing and verifying work completion in their respective areas of responsibilit _ _ _ _ _ _ _ _ _ _ _ _ _ _ , _ _ - _ - _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ _ _ , ! l In September 1987, the senior system specialist position for air systems l and certain other balance of plant mechanical systems, e.g., auxiliary- I off gas, demineralizers, etc. , was filled by a contractor engineer. That .) individual was tasked with reviewing the overall. status of the material { condition of the " owned" systems and to recemmend actions to improve 1 overall system operation and reliabilit With respect to the PNPS I instrument and service air systems, a comprehensive study was conducted as l documented in an office memorandum SG-87-384, from D. Rosinski to W. Clancy dated October 27, 1987, " Air System Status . Report." The PNPS study was conducted as part of the licensee's review of NRC IE Notice 87-28 " Air Systems Problems at U.S. Light Water Reactors," which attached i NRC case study report AE00/C701, " Air Systems Problems at U.S. LWR's." l The NRC cace study report reviewed a large number of problems experienced ! by U . S . light water reactors due to air system related failures or system l degradations, identified eight root cause deficiencies and included five !' recommended actions', The PNPS study, which included review of applicable current industry and ; i regulatory standards for air systems,- vendor technical manuals and existing maintenance and surveillance procedures, addressed three of the eight NRC case study root cause deficiencies including: (1) mismatched equipment-the air quality capability of the installed filters and dryers do not match the design requirements of the equipment using the air; (2) ! preventive maintenance - system maintenance is not always performed in accordance with manufacturer's recommendations; and (3) air quality - air quality is not monitored periodically to assure that system dryers and filters are working properly. The PNPS air system status report identi-fied the specific PNPS status and included specific recommendations with respect to the three root cause deficiencies described above. Overall, the report concluded that the PNPS air systems are in poor material condi-tion and the preventive maintenance program does not meet the current NRC recommendations. Corrective actions to implement the suggested NRC guide-lines and improve overall system performance were divided into four short-term and seven long-term items, which collectively represent a j substantial commitment to improving the material condition of the air j system i The licensee's actions taken in response to IE Notice 87-28 are being reviewed separately as part of the ongoing NRC resident inspection activities, and the delineation of specific recommended actions and their schedule, as addressed in the PNPS status report, is not considered within the scope of the AIT repor However, the team again notes that the i licensee's study was comprehensive and detailed and included sound recom- j mendation Also, as a result of the system specialist's extensive { involvement in conducting the study and developing the report, substantial system knowledge was acquired that contributed to the overall licensee recovery actions to the losses of the instrument air system on November l 12, 198 I - - - - - - - - - - - - - - - - - - - - - _ _ _ _ _ _- i l Based on the PNPS systems group status report recommended actions to improve overall air system operation and reliability, and in specific consideration of certain lessons learned during the loss of offsite power event, on November 16, 1987 the systems group initiated an engiraering evaluation request (EER), " Station Air Systems Deficiencies." ine EER highlighted in summary that the instrument and service air systems require upgrading and presented a detailed description of typical problem areas and recommendations that require further corporate nuclear engineering evaluatio The problem areas and recommendations addressed four short-term (pre-startup) items regarding (1) the inadequacy of the existing temporary system (TM 87-30) for connecting a portable air compressor to the station air system; (2) the existing design adequacy of air heade . isolation (block) . valves upon loss of air pressure and subsequent i restoration, (3) air dryer and filter sizing for current system loading and operational transients, and (4) clarification of air quality specifi-cations and testing requirements. Six long-term (post-startup) items were identified for engineering evaluation including: (1) use of temperature control . valves in the compressor cooling system, (2) installation of a central base / trim load station to provide centralized control and system status, (3) adequacy of having air receivers placed between the compres-sors and air dryers, (4) non corroding intake piping installation, (5) the need for the installation of d/p gages across filters and air dryers to monitor operation and determine need for replacement, and (6) the need for an in-line dewpoint senso In addition, the EER identified the current priority of the 40 identified work items related to the air systems, 30 of which are considered priority one (restart). The EER was in the onsite management review and approval process at the time of review by the AIT team, and was subject to in process revision prior to approval by the nuclear operations department manage Nonetheless, the EER demonstrated the systems group awareness of the importance of correcting known or perceived station air system deficiencie .3.5 Conclusions and Recommendations Initial and subsequent operator actions in response to the losses or degradation of the instrument and service air systems, as a result of the loss of off-site power event on November 12, 1987, were properly conducted and were in accordance with an approved facility emergency procedur Although procedure No. 5.3.8, " Loss of Instrument Air", was not considered deficient for the events that occurred with the plant in a cold shutdown condition for an extended period, the AIT team members noted that the l procedure lacks specificity of subsequent operator actions. Other l improvements or changes may be warranted, based on a planned licensee ' review of the procedure for its overall adequacy for performance capabi-lity while the plant is in a normal operating condition, and based on i i ricorporation of appropriate revisions to reflect air system design I changes or modification I i - - . - _ _o _ _ _ _ - _ _ 1 ( i The instrument air system non-availability that occurred on November 12, 1987, never threatened the ability to maintain the reactor in a safe, controlled cold shutdown concition. Further, air system header isolations and component actuations occurred as designed, as verified by plant operators and system engineer The extended loss of the instrument air system became a nuisance, but not a detriment, to operator performance l during the loss of off-site power condition, as compensatory measures were required to monitor certain tank or water bay levels or to monitor and pump various plant sump ! Recovery actions for restoring air system pressure were appropriately cautious and methodical, with proper consideration of the stable, non-challenging reactor and plant condition Operators started the only available air compressor within one hour of the loss of of f-site powe Day shift recovery actions benefited from the systems engineer's knowledge of air system design and material conditio Operations, maintenance, engineer and other support activities were well integrated into the over-all recovery ef fort Sound management decisions for recovery actions were based on well understood plant conditions, operational needs and air system problems. Contingency plans and schedules considered realistic and periodically updated time estimates fur equipment maintenance, delivery, temporary modification and testin Extensive air system deficiencies had been identified during a recent , comprehensive review by the systems group. The review concluded that the i PNPS air systems are in poor material condition and the current preventive j maintenance program does not meet the current NRC recommendations. How- j ever, substantive recommendations were made to escalate the priorities of j previously identified and additional maintenance activities and to conduct i engineering evaluations of identified problem These activities are being conducted in support of the utility management assessment of restart readines Licensee efforts to upgrade the existing design and material condition of the instrument and service air systems, and to improve the loss of instru- I ment air procedure should continue as planned. No additional NRC recom-mendations are considered necessary as a result of the AIT review and assessment of the November 12, 1987 loss of off-site power even .0 Human Performance 5.1 Organization l l The organization established by Boston Edison Company for the normal ] operation of Pilgrim is divided into the Nuclear Operation Organization j and the Nuclear Engineering and Quality Assurance Organization. The ' Nuclear Operation Organization directs operational activities at Pilgrim while the Nuclear Engineering and Quality Assurance Organization provides i onsite and offsite suppor Both of these organizations report to the Senior Vice President, Nuclear who reports directly to the Chief Executive Office ! j ! l _ _ _ _ ._ _- -- .- . _ _ , ! In addition to normal operations, these organizations are staffed and qualified to take those emergency actions necessary to implement the Emergency Plan including immediate protective measure Pilgrim Station is operated by the Nuclear Operations Department (NOD) of the Nuclear Operations Orgainzatio The NOD Manager serves as the Station Manager and, as such, is responsible for the overall safe opera-tion of Pilgri The NOD Manager reports to the Vice President-Nuclear Operation (The position of Vice President - Nuclear Operations is temporarily being filled by the Senior Vice President, Nuclear). Within NOD, there are five (5) areas of responsibility. Each area is headed by a Section Manager who reports to the NOD Manage The Operations Section, headed by the Operations Section Manager, is responsible for the safe and proper operation of the plant. The Section Manager's immediate assistant for operational activities is the Chief Operations Engineer (COE). The immediate on-shift position of authority to carry out plant operations responsibility is the Nuclear Watch Engineer (NWE), who oversees all shift operating personnel. The Nuclear Operating Supervisor (NOS) functions as an assistant to the NWE and in the NWE's absence, assumes the responsibilities of the NW The Nuclear Plant Operators are responsible for the manipulation of controls as necessary to perform plant operations. The Nuclear Auxiliary Operators are responsible for performing component and/or system operations outside of the Control Room. The Nuclear Plant Operators and the Nuclear Auxiliary Operators are responsible to the NOS and take direction from the NOS or the NWE. Other on-shift operations personnel include a Radiochemistry Technician and a Shift Technical Advisor (STA). The Maintenance Section, headed by the Maintenance Section Manager, is responsible for the maintenance of the plan At this time, the Main-tenance Section Manager holds the additional responsibility of Outage Manager. Maintenance Section responsibility includes the periodic preven-tive and corrective maintenance of plant systems and equipment, the day-to-day general upkeep of onsite facilities and the procurement of parts and supplies to support these activitie The Maintenance Section provides on-shift maintenance personne The Technical Section, headed by the Technical Section Manager, provides engineering services and plant analysis support at the Statio This Section provides many of the resources to staff the Technical Support Center (TSC) in the event of an emergenc The Radiological Section is responsible for the implementation of the Pilgrim radiation protection and radiological / environmental monitoring program The Radiological Section provides one or more Radiation Protection Technicians on each shif The Administration Section is responsible for station personnel health and administrative service I i l _ _ _ _ . _ - _ _ _ _ _ _ _ _ _ _ _ _ - _- __ __ - __ ___ The Pilgrim Emergency Response Organization is composed of pre-designated I station and corporate personnel trained to augment shift operating ) personnel in an emergenc The normal shift operating group provides the ' initial response to an emergency. This group is trained to be self- ! reliant for a sufficient amount of time (30 to 60 minutes) for personnel assigned to the Emergency Response Organization to assemble and integrate i smoothly into the emergency respons ' The positions of Emergency Director and Emergency Coordinator provide the direction and coordination needed for an effective emergency response. The Emergency Director is responsible for the direction of all onsite emergency response activities. The Emergency Coordinator is responsible for the overall coordination of the BECo emergency response effort and for the coordination of that effort with the activities of offsite support agencie l In all emergencies the NWE is responsible for initiating the implementa-tion of the Emergency Plan. When an emergency situation is initially identified, the NWE will classify the event and assume the role of Emergency Directo An emergency may be declared anytime that, in the judgement of the NWE, the plant status warrants such a declaration. The initial emphasis of the Emergency Director is to assess the emergency situation, make required notifications to offside agencies and call out the Emergency Response Organization, if needed. The NWE will continue as Emergency Director until relieved. The NWE will brief the incoming Emergency Director on the emergency classification, plant status, radiological conditions and protective measure Once briefed, the incoming Emergency Director will relieve the NWE of all Emergency Director responsibilities. Command and control of the emergency will then transfer to the TSC once that facility has been activated. The Emergency Director reports to the Emergency Coordinator. The Emer-gency Director is normally the Station Manager, the Technical Section Manager, the Operations Section Manager or the Chief Operating Engineer. The Emergency Director is required to be an ANSI 3.1 qualified manager / supervisor or a licensed senior reactor operator. The Emergency Coordinator is normally the Vice President-Nuclear Opera-tions. The Emergency Coordinator is required to be an ANSI 3.1 qualified manager. The TSC Supervisor and staff are responsible for providing technical support to the NWE in mitigating accidents and their consequence The TSC staff provides the operating staff in the Control Room with in-depth diagnostic and corrective engineering capabilities while relieving them of peripheral duties and communications not directly related to reactor system manipulations. The staff is able to analyze current and projected plant conditions and, through communications with the NWE, provide l I ) technical support and recommendations regarding emergency action The 3 TSC staff is composed of several engineering and technical disciplines such as reactor engineering, plant analysis, operations, chemistry, main-tenance, instruments and controls, electrical engineering and fire l protectio ! 5.2 Event Classification and Notification i The Emergency Plan provides four emergency condition classifications. In ascending order of severity these classifications are Unusual Event, Alert, Site Area Emergency, and General Emergenc The first two classi-fications involve degradation of the safety of the plant and initiate precautionary measures should the situation become wors For loss of power conditions, an Unusual Event is declared if there is irradiated fuel in the vessel and RCS temperature is greater than 212 F, and (a) there is a loss of all onsite ac power capacity; or (b) there is a loss of all of f site power (i .e. loss of line Nos. 342 and 355, and shutdown trans-former). For loss of power conditions,' an Alert is declared if there is irradiated fuel in the vessel and RCS temperature is greater than 212 F, and (a) there is a loss of all onsite dc power for 15 minutes or less, or (b) there is a loss of all offsite power coincident with loss of both emergency diesel generator On November 12, 1987, shortly after the LOOP, control room personnel reviewed emergency action levels (EALs) prescribed in the Pilgrim Emergency Plan. Based on their understanding of plant conditions at the time, they concluded that no threshold had been reached requiring actions prescribed by the Emergency Pla In accordance with the provisions of 10 CFR 50.72 Immediate notification requirements for operating nuclear power reactors, control room personnel notified the NRC Operations Center via the emergency notification system (ENS). This notification was made at about 2:55 a.m. (about 50 minutes after the LOOP). The licensee reported a scram signal on loss of offsite power; 40-50 mph winds with snow; the plant was stable in cold shutdown; and the diesels were manned, loaded and carrying house load Licensee personnel onsite made additional notifications and took action to supplement onsite resources as summarized below: - Control room personnel contacted the Nuclear Operations Department Manager at his home at 2:20 He arrived onsite at about 3:00 Between 3:00 a.m. and 4:30 a.m. the Nuclear Operations Department Manager notified the NRC Resident Inspector the Senior Vice President-Nuclear, the BECo Public Information representative, and < the Middleboro State Police (local Civil Defense). The Nuclear Operations Department Manager also attempted to reach the Chief Operating Engineer; however his phone was inoperable due to the stor _____-_____-___-_____- - _______ - _ - _ - _ _ _ - =- - . _ _ - - - I l - At about 2:35 a.m. the Assistant Director of Outage Management who I was on shift contacted the Maintenance Section Manager. The Main-tenance Section Manager arrived onsite at about 7:30 At about 3:20 a.m. the Senior Electrical Engineer in the Maintenance Section was contacted. He arrived onsite at about 4:00 Between 4:00 a.m. and 5:00 a.m. the BECo Emergency Preparedness organization made notifications to Community Representatives for the towns of Marshfield, Duxbury, Kingston, Carver, and Plymout Taunton and Bridgewater were notified later in the da At about 5:30 a.m. the BECo Emergency Preparedness . organization notified the Commonwealth of Massachusetts, Departments of. Public Health and Civil Defens Community Representatives and Commonwealth representatives were again notified when the "B" Emergency Diesel Generator was shutdown and again when normal power was restored to Pilgri [ Note: The NRC Resident arrived onsite at about 5:00 a.m. He established telephone contact with Regional and Headquarters Manage-ment to apprise them of the situatio Periodic briefings by the resident inspector were continued throughout the even After some initial telephone difficulties, an open line was maintained between the NRC Operations Center and the Pilgrim Control Room commencing at about 9:00 a.m.] As determined during AIT interviews, licensee personnel re-evaluated the emergency classification of the event at various times during the recovery effor The Nuclear Operations Department Manager stated that he re-evaluated the emergency classification and EALs af ter he arrived onsite and again when the "B" EDG was taken out of servic The day-shift Nuclear Watch Engineer stated he also re-evaluated the event classifica-tion after the "B" EDG was taken out of servic TSC supervisory personnel also stated they evaluated the event classificatio These individuals believed that an " Alert" should be declared if the "A" EDG was ! lost (after the "B" EDG was shutdown). Based on criteria in the Pilgrim Emergency Plan, an Alert would not be required for a complete loss of all offsite power coupled with a loss of both diesel generators unless RCS temperature was above 212 .3 Use of Support Organizations and Personnel , i The management of support organizations and personnel was examined from ! four aspects. These were 1) immediate support to the operating organiza-tion af ter the occurrence of the LOOP; 2) NED support; 3) use of the Technical Support Center; and 4) use of the Maintenance Sectio _ _ _ _ _ _ _ _ _ _ _ _ The licensee initially supplemented the plant operating organization with additional personnel to deal with the recovery from the LOOP even Once the plant had been stabilized and initial notifications made, the Station Manager and Senior Electrical Engineer were recalled to the sit These two individuals arrived at about 3:00 a.m. and 4:00 a.m., respectivel Recall of other resources was not undertaken by the Station Manager or the NW In light of the plant conditions, the severe weather, and the impending arrival of the daytime work force this decision appears to be appropriat The NED support was on an informal basis until the Technical Support Center was staffed at about 1:00 p.m. Early in the recovery effort, Prior to staffing the TSC, it appears that NED management viewed the event as essentially a plant operational matte Interviews by the AIT indicated that numerous staff-level conversations occurred between NED personnel and onsite system engineers, technical personnel and various Maintenance Section individual Most of these conversations concerned electrical engineering issue One of the significant informal contacts between the NED and the site concerned the "B" Emergency Diesel Generator (EDG). During the morning of November 12, an electrical engineer in the Power Systems Group discussed the apparent failure of the "C" phase ammeter with onsite engineers from the Electric Lab. This type of discussion is typical of routine technical advisory discussions between NED and the site, according to the NED Manager. In this instance, NED management was not involved in discussions that led to the shutdown of the "B" ED At about 1:15 p.m. , NED established an open line to the Technical Support Center (TSC). By 2:15 p.m., the Power Systems Group established 24-hour coverage to support TSC requests, and a representative of NED was onsite in the TSC. From this time until the recovery effort was completed, NED provided constructive support to the station via the TS Activities included safety evaluations of temporary procedures and assessments of several temporary modifications being considered by the site. One of these modifications, which was not implemented. in part on NED recommenda-tion, related to providing vital power to one of the non-vital air com-pressor A major support activity involved the " administrative staffing" of the TSC. The TSC was not formally activated pursuant to the Pilgrim Emergency Plan; however at about 12:45 p.m., the station manager directed the technical section manager to administratively staff the TSC in support of the'NWE. Several individuals had already been working on an ad-hoc basis in the TSC reviewing plant condition The TSC was staffed by about a dozen individuals and logs were established on November 12, 1987 at 1:08 p.m.. The positions of Emergency Director and Technical Support Center Supervisor were assumed by the Technical Section Manager and Technical Group Chief Engineer, respectivel No offsite aspects of the Emergency Plan were deemed necessary. Operations - ____-_-_- __ Center " administrative support" was provided by the Maintenance Section Manager and Maintenance Section personnel. The TSC remained staffed until 3:45 a.m. on November 14, 198 The TSC carried out a number of actions in support of the NW These actions included: - Monitoring and tours - of the plant to ascertain sump levels; moni-toring of water accumulation and subsequent observation of the decontamination of the corridor area on the - 1 foot level of the Process Building (See Section 5.7). - Monitoring of fire protection system components to assure that no degradation was caused by the loss of station ai Review and approval of various temporary procedures and temporary modifications. These actions included Operations Review Committee .(0RC) reviews required by Technical Specification Monitoring of station battery statu Support to the Operations Section and Maintenance Section by monitor-ing backscuttle activities, diesel generator surveillance and system restoration after power was restore Interviews by the AIT indicated that, although the TSC was effective in carrying out its activities, the role of the TSC may not have been clearly understood by the Control Room personnel. The procedural aspects of administratively staffing the TSC are not clearly delineate Addi-tionally, provisions are not made for the management of certain recovery actions from the TSC under circumstances where the TSC is not formally activate The last major support activity involved the use of the Maintenance Sectio Under the direction of the Maintenance Section Manager, this section implemented the actions necessary to troubleshoot equipment, ; correct faults and return components and systems to an operable stat Major activities included washdown of switchyard insulators, restoration of the "B" emergency diesel generator, restoration of the Startup Trans-former, connection of a temporary high pressure air supply, and removal of main generator disconnect links to allow backscuttle of offsite powe ' The Maintenance Section Manager provided much of the coordination and direction of the recovery activity. This activity was somewhat delayed in getting started as the Maintenance Section Manager did not arrive on site until 7:30 a.m. , many work force personnel were delayed in reaching the site by the adverse weather, and management and supervisory personnel needed to assess the recovery options. Starting at about 10:00 a.m., periodic meetings were held in the Maintenance Section Manager's Office to assess recovery progress and provide direction and coordination of site personnel activities. Attendees included the Maintenance Section Manager, Nuclear Operations Department Manager, Executive Assistant to the Senior l ! . -- - . _ _ . I - _ ___ . __ _ _ _ - _ [ s! , ! ' : Vice president-Nuclear, Technical Section Manager, Assistant . Chief Main-tenance engineer and others. Interviews by the AIT indicate that no one : from the Operations Section or NRC attended these meetings. The BECo l recovery schedules prepared by the Maintenance Section Manager are presented in Appendix These represented the status and decisions from ! these periodic meeting l The periodic meetings appeared to provide focus and direction to the LOOP recovery ef fort. The absence of the Operations Section may have impeded a a ,) full understanding of recovery actions by operations personnel and meant that' planners may not have fully been able to consider operational aspects of their decision .4 Management Activities and Communications The AIT examined various aspects of overall management activities and-communications as they related to the LOOP event. These aspects included - overall control and coordination of activities, planning actions (before the event and during the recovery), compensatory measures taken in ,d response to changing conditions, control room activities and use of the 1 TSC. Also reviewed were management aspects of the backscuttle, the "B" ' ., l emergency diesel generator shutdown and the loss of station ai ! Overall control of the plant in immediate response to the event and ! initial actions to notify ar.d recall support personnel were adequate, as ! described in Section 5.2. Interviews by the AIT indicated that the role ' and responsibilities of the NWE remai r.ed clear and consistent with j established practice and procedure The NWE's clearly understood and t l properly observed requirements to assure reactor safety at all times ! during the event and subsequent power recovery activitie ; Overall management of the recovery effort appears to be somewhat frag-mented over tim It was difficult for the AIT to determine who had overall responsibility for managing the recover , Early morning efforts ) by Maintenance Section personnel focussed primarily on the 345 kV switch-yard without clear management involvement or consideration of the back- ! scuttle pat In general, the Nuclear Operations Manager remained in the control room, except when attending meetings in the Maintenance Section Manager's office. His operational management involvement was evident, ,' ! however, his overall direction of station activities seemed less clea I By late in the morning on November 12, management of the recovery effort appeared to reside primarily with the Maintenance Section Manager. Strong i direction and coordination from the Maintenance Section Manager was ,j evident throughout much of the remainder of the recovery, as exemplified l by the periodic meetings he chaired e.nd his issuance of BECo recovery 3 document The overall recovery appears somewhat lengthy; however, this may be attributed, in part, to the weather, the cold shutdown condition of the plant (which did not mandate speedy action) and BECo management's stated directive to proceed slowly and deliberately during the recover \ l - _ . _ _ . _ _ _ _ _ _ - _ _ _ - _ - _ _ _ . _ _ _ _ . _ _ _ - - . _ _ _ _ _ . _ . . - . . , ,, 4 i , .8 s g . t Planning activities prior to the event were limited. It is not clear that BECo examined the overall impact of having significant c'omponents for a - large number of systems out of service, particularly in the light of the possible occurrence of analyzed events. For eka).ple, t.aking the shutdown transformer out of' service for an extended peribd end not removing the i main generator disconnect links reduced the st'ation's flexibility in vapanding ' k a LOOP event. Having several air compressors simultaneously out of' 'supice also reduced flexibilit ' ' Planning fo .he poss[oiiity of severe weather appeared limited and may be somewhat dependent or good practice or previous experience. BECo does have a high winds procedun for dealing with the potential for winds in excess of 75 miles pe) hou Planning activities dur'ing Om reconry were slow sn getting start.ed; gh owever the plannira actions by tbe Maintenance Section were a positive i dup ,iin bringing focus to the recovery ef fort. Hourly meetings in the Maintenance Section Marager's office were held to assure coordinatio Planning for the demands of an extended event on personnel fatigue was ' generally acceptable. For example, the Maintenance Section Manager used a computerized data base to mcnipor ami adjust personnel work hours and the Operations Section maintained 'normsi control room watch rotation qq y Several planning weaknesse- during the event were noted by the AIT. It was not clear that management, was moving aggressively to assure restora-tion of power capability it a timely manner. For example, the initiation of the backscuttle effort was not evident until mid-day on November 1 Another example, involves thet 'B" emergency diesel generator. When the cause of the failure of the current transformer circuit was determined, efforts could have moved more promptly to restore the emergency diesel generator. Another planning weakness contributed to poor communications between the operations and maintenance organizations. As described in Section 5.3, Operations personnel were not included in the Maintenance Section Manager's periodic planning meeting Planning and control of insulator washdown was hampered by conflicting prioritie Initial washdown efforts were directed towards the path between the No. 355 'ine and the startup transformer (the shortest wash-down path); however, it is not clear that selection of this path con-sidered the availal equipment necessary{ li'.y of insulation to restore resistance the startup transformer andto high potential operation andtest the necessary test c' ' checkout time requirements. Later washdown efforts were redirected towac s the path between the No. 355 line and the main transformer (the longer washdown path). Coincidentally, weather condi-tions improved and the washdown of the longer path was completed in a shorter time than the washdown of the path to the startup transforme Planning activities during the recove ry included consideration of the status of station air and the actions necessary to obtain a tempora ry diesel driven air compressor and connect it to plant instrument air system Implementation of this installation was delayed by the onsite delivery of the air compressor; recovery of normal power supplies to station air compressors precluded the need for this installatio _ - _ _ _ _ _ _ _ _ - . ', . . , . . Ml % .. % '# vlp J - q ., - v ' ' ./ - 6 , N' Lt ; , y < ^ t ,, , 4 ' 'i Management of Na bachscuttle activity say hanpered by the presence of , ,' , ' ' numrous tags 4 fectinj_.tbe an f li ary t ransformer . and main generator iso phase buequick discenrg;t'lGks. Some of theaf tap were placed in ' support of the pending blaceut- dieself generator installation and some were left ever' tags from cenpletsj maintenance actions. p Strteter control ' of tags may, have. : reduced the ~ effqrt necessary tc cieu the tags at a ' { critical tim .,, I Compensatory hueesures jn:pinentid by the 1:arsee dring the LOOP and ' recovery were satisfer. tory. Examples incluN , - Personnel monitur"ng'of emergency diesel ggnerators , - .fh.nitoring of sudp , . - Plytt tcurs and inspections - Fire watcnes to compensate foc > fire suppression system realignments ; ' made as a result of the loss 'of station air to certain fire suppression system valves ' . Activities in the control room and the TSC were gene-tity well managed by licensee supervisory and management personnel. Interviews by the AIT d, indicated that control of personnel and communication 7s were generally '/ effective with two exception The first concerne,i the number of L ! personnel in the control room. The continuity, pesence of management and L NRC personnel in the < NVP s c f:fice, in part, t ocessitated because of the backup telephow. comr:u r'pattarkthat were established with the ENS line B initially incpdrable , 4 %rced the NWE cut of his office and may have preventd g nth hom mmW ef fectively managing watch section activitie < Tli seconci control and con;munications weakness involved the shutdown of the "B" enegancy diesel generator (EOG). Although the COE and NWE were . both made ~ aware of'1 % "C" phase current transformer problem shortly I , before the EDG ws stsped 60 '11:35 a 4., they both lef t the control room ' / ar,a critical- time without clearly ascertaining the intentions of the NOS regarding the EDG. ThvCOE and NWE did not give sufficient attention to a vital power supoly an0 alternatively, both pursued tasks of lesser importance in /olvi ng the air compressors and turbine building closed cooling weep ' Coincidentally, the Nuchtar Operations Department Manager, who had been present in the control room much of the morning, also was absent. Consequently, the leadership in the control room was not suffi-dont to assure continuation of this v%al power source until a more , mWdered decision regarding '!ts shutdowa could have been mad c Licensee managemeat acted effectively and responsibly in administratively Oaffing the TS As discussed in Section 5.3, this contributed to ef fective management of tu fecovery. Interviews by the AIT indicate that the Mcensee believes the TEC could have been of greater value had it been staffed sooner. One a wect' of' the TSC staffing is noteworthy from a management standpoint. The hclear Operations Department Manager would normally be stationed in the TSC if the TSC had been activated pursuant to _ _ _ _ _ _ _ _ - _ _ _ - _ _ n; the Emergency Plan. In that the administrative staffing of the TSC is not prescribed by the plan, the Nuclear Operations Department Manager remained in the Control Room. Had he been in the TSC he may have been able to more effectively direct the overall recovery effor At various times during the recovery effort the licensee made statements to NRC to the effect that " Power can be restored, if really needed, in a very short time." AIT interviews indicate that statements of this nature were generally without clear technical basi Initially, power restora-tion was precluded by severe weather and lack of clear knowledge of conditions in the switchyar Later, power restoration via the startup transformer would have been precluded by the need to properly test and checkout the transformer. Essentially, all knowledgeable personnel stated to the AIT that they would strongly recommend complete checkout of this transformer before attempting to place it in servic Power restoration via backscuttle was not immediately possible until disconnect links were removed, tags cleared, and insulator washdown was complet .5 Operator Action The AIT reviewed actions of both licensed and non-licensed plant operators in response to the LOOP and during the subsequent recovery. No signifi-cant discrepancies were identifie Appropriate concern for reactor safety and procedure adherence was maintaine .6 Procedure Adequacy The AIT conducted a general review of portions of selected plant proce-dures that were of interest to this event. Interviews by the AIT revealed that several procedures are being reviewed by BECo to assure that they are . adequat These procedures are: - 2.4.16, Distribution Alignment Electrical System Malfunctions - 2.4.144, Degraded Grid Voltage - 2.2.36, Instrument Air System - 2.2.37, Service Air System - 2.2.46, Control Room / Cable Spreading & Computer Room Heating, Ventilation and Air Condition system - 5.3.8, Loss of Instrument Air - 5.2.2, High Winds (Hurricane) 5.7 Radiological Aspects The AIT reviewed the radiological aspects of the LOOP and interviewed the Radiological Section Manager. The AIT also inspected certain radwaste areas in the Process Building affected by a minor water flooding even __ _________ ___ _ _____ -- - -- _ _ _ . _ , Early in the recovery period the plant staff recognized that the LOOP prevented operation of various radwaste system components such as sump pump They also recognized that the loss of station air affected operation of many air operated valves in the radwaste system. Radio-logical Section, Operations and Technical Support Center personnel made a ' number of plant tours to monitor systems and components for leakag Particular attention was paid to sumps because leakage was accumulating therein and could not be readily remove { At' 2:45 p.m. on November 12, plant personnel discovered minor flooding on the -1 foot elevation of the Process Buildin This was caused by water backing up through the drain system and overflowing from the turbine building sum At about the same time plant personnel discovered the ' ingress of ground water through seams and a conduit penetration in the West wall of the Process Buildin This water was accumu'tating on the floor of the Monitor Tank Room. The principal source of this water was from a conduit penetration, which contained a conduit leading out to a small sump for the switchyard area (adjacent to the switchyard gate). The failure of the switchyard sump pump coupled with excess ground water in the switchyard (due to insulator washing and melting snow) resulted in this conduit floodin In response to the minor flooding, which generally was no more than " puddling" on some of the floors in the radwaste area and in the radwaste area hallway, the plant staff placed a temporary sump pump in the' turbine building sump. This pump was powered by vital power and was used to transfer the excess water to the Chemical Waste Tank. Radwaste Section personnel successfully controlled the water ingress and sump overflow throughout the event. Site personnel estimated about 3000-4000 gallons were transferred to the Chemical Waste Tan When power was restored, normal radwaste processing was re-established and the floor of the radwaste area hallway was decontaminate (Initial contamination levels ranged from 1000-5000 dpm/100 cm2 ). Decontamination was completed within four hours. No airborne contamination was identified and no personnel contamination occurred during or after the LOOP even No other significant radiological issues were identified by the AI Interviews by the AIT revealed that Pilgrim has had some previous diffi-culties with ground water ingress, both through seams in the Process Building wall and through the conduit penetration for the switchyard sump pump. This is caused, in part, by a high water table around the Pilgrim Station and by insufficient corrective action to deal with identified sources of leakage. This water ingress contributes to a radwaste proces-sing burde Licensee personnel indicated that BECo plans to more , aggressively pursue courses of action to mitigate this proble . _ _ _ _ _ - - _ - . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ -__-_ __- _ _____ __ _ _____ i k 59 .; I 5.8 Conclusions and Recommendations The following. conclusions are made regarding human performance during the LOOP and subsequent recovery: - Operational response to the LOOP, event classification, notifica-tions, compensatory measures, radiological control activities, and operator actions to assure reactor safety were correct and in accordance with procedures and requirement Overall management of the recovery effort was somewhat fragmented and unclea Communications between organizations and between individuals resulted in uncertain roles and some undirected action Administrative staffing of the TSC and formal recovery planning activities were positive steps in providing focus and direction to the restoration of normal power. Procedural guidelines do not exist for the administrative staffing of the TSC. - Certain procedures governing degraded plant conditions (e.g., loss of power and loss of instrument air) may not be sufficient to clearly guide recovery actions from events of this nature. The following recommendations are made to strengthen human performance at the Pilgrim Station: - Strengthen the management guidelines and roles to assure a clearer and more positive direction to recovery activities following major events. - Strengthen communications practices to assure clear understanding and directed actions. - Re-evaluate emergency action levels regarding loss of onsite and offsite power for situations where fuel is loaded in the reactor vessel and RCf temperature is less than 212 < - Develop and implement procedural guidelines for administratively staffing the TSC to support the operating organization in situations where Emergency Plan activation is not appropriate. - Re-evaluate plant procedures that may be required in loss of power events to assure they are sufficient to guide recovery from degraded plant condition Although not directly related to human performance, BECo should j continue actions to reduce ingress of ground water to the Process j Building and consequential radiological burden. - The practice of having the main generator bus quick disconnect links installed during an outage should be reviewed relative to the recent j even j - - - - -- J 60 ! - The plant configuration and equipment out of service for outage maintena'nce created operational inflexibilities during the event which, under other circumstances, could have had serious impacts on the operator's ability to cope with the situatio BEco should provide a review process for plant configuration and equipment in a maintenance status to assess plant and operator needs during outages to cope with outage operation oriented transients ( loss of offsite power, loss of shutdown cooling). 6.0 Safety Assessment The safety implications of this event have been assessed for the actual ] occurrence as well as for the event under different reactor conditions 1 (decay heat level) and as a station blackout precurso .1 Reactor Safety Significance of Actual Event At the time of the LOOP, Pilgrim Nuclear Power Station (PNPS) had been shutdown for an extended outage for some 19 months. The plant was in a cold shutdown condition with the reactor coolant system temperature less than 212 F and reactor coolant pressure maintained at about 2 psig with the reactor vessel vented. Reactor core decay heat level was minimal due to the extended shutdown time and with the core being comprised of ore third new fuel. The decay heat was insufficient to heat the coolaat to a temperature necessary to perform pre-startup hydrostatic testing. The RHR system was being operated in a recirculation mode wher'ein heat was being added by RHR pumps to the coolant to maintain temperature at about 180 Reactor vessel water level was up at the 212 in. level well above the normal operating range. All four RHR pumps were available for service, only."A" RHR pump was running. Only one offsite power circuit through the stcrtup transformer was immediately available and all plant safety systems were powered from that sourc Both emergency diesel generators were available in standb A discussion of key system and equipment status prior to the event is provided in Sections 3 and The loss of offsite power resulted in both a reactor protection system (RPS) actuation and primary containment isolation signal (PCIS). The latter caused isolation of the RHR which was restored after 39 minute During the next 24 hours, RHR was lost (isolated) two more times, once for 23 minutes and again for 14 hours 27 minute The longest interruption of RHR occurred when the B diesel generator was taken out of servic RHR wasn't restored until after offsite power was restore Because decay heat levels were so low, reactor coolant temperatures did l not increase during the course of the event when RHR was not available. l In fact some recorded temperatures decrease Average temperature ] decreased about 2 Heat losses through the vessel contributed to this ' l , reactor cooldow In general reactor coolant and vessel metal I temperatures remained around 170 F-180 F. Thus, at no time during the event, which lasted more than 24 hours, did the conditions within the reactor coolant system degrade to a potentially safety significant condition (i.e., coolant temperature was always less than 212 F). l (. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _________i _ _ - _ - _ ___ _ _ When the B diesel generator was taken out of service the operability of two other systems was affected. These were the instrument air and normal control room heating, ventilation, and air conditioning (CRHVAC). Air operated instrumentation was lost when MCC B14 which powers compressor K104B was de-energized. The loss of these non-safety systems should not be particularly troublesome since no credit is taken in the plants final safety analysis (FSAR) .for the availability of either system during an accident. The loss of air pressure causes operability problems with some systems used during normal plant shutdown and in particular CRHVAC requires instrument air for proper chiller and damper operatio i However, the emergency air filtration system was operable, and occasionally used, during the even It is not known if operation of CRHVAC was affected when air pressure was available but low during the even Control room heat loads were such that somewhat uncomfortably high temperatures were felt by operations and NRC personnel present raising a concern about control room habitability and control room equipment operating temperature specifications (see AE0D C604). Because at least one diesel generator was operable at all times, control room heat loads during this event may have exceeded those expected during events involving loss of all AC power which would result in the loss of both safety and non-safety control room ventilation system It is believed that the control room temperature did not rise much above 90 F. The maximum operating temperature specification for equipment located in the control room is 120 F. The availability of the emergency air filtration system and the cool outside temperatures combined to help keep bulk control room temperatures well below equipment operating temperature specification FSAR analyses predict a maximum control room temperature of 114 Computer equipment cabinet temperatures were recorded on the process 4 computer alarm printout which shows local temperatures of about 95 F at 3:04 a.m. and up to 108 F at 9:41 a.m. (about 7 hours after LOOP). By 4:45 p.m. cabinet temperatures were close to or below the alarm set point of 95 F. It is not known if these or other electrical equipment cabinet temperatures exceeded the highest recorded values. It is also not known how indicative these recorded temperatures are for other cabinets contain-ing essential safety equipment. No equipment failuras were reported which could be attributed to high temperatures. Therefore, the actual impact of any temporary or sustained loss of CRHVAC which may have occurred was ni .2 Reactor Safety Implications at Hiaher Decay Heat Levels In light of past history at PNPS it is possible that the LOOP event of November 12, 1987 could have occurred either while the reactor was at power, or if shutdown, with substantially higher core decay heat levels than existed at the time of the event. If at power, a reactor trip would have occurred with the feedwater system and condenser lost for decay heat remova High Pressure Coolant Injection (HPCI) and Reactor Core _ - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ - _ _- _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ I Isolation Cooling - (RCIC) systems would have been used for core cooling with steam being relieved to the suppression pool. Under these circum-stances the plant could remain on RCIC with Suppression Pool Cooling (SPC) for a considerable period of time. RHR could also be initiated when the reactor pressure and temperature had been reduced. Decay heat removal should not be significantly affected with a long loss of offsite power and no other system failure During the LOOP event of November 12, 1987 at least two other noteworthy malfunctions occurred. The first being degradation and then loss of plant air systems. This malfunction could have affected CRHVAC and caused other BOP systems to be inoperable. Other than being a nuisance it does not appear to have affected any safety systems capability adversely. The second malfunction involved the B diesel generator current transformer (CT) circuit and its subsequent removal. from service. The removal (or failure) of either diesel generator results in the de energization of certain vital safety buses which results in PCIS, in particular a group III isolation of the RHR suction line Thus, just as RHR was lost several times for varying durations on November 12, 1987, it could also have happened if the reactor had been shutdown more recently with higher decay heat levels given a LOOP and one diesel generator failur The reactor coolant heat up characteristics would have been different from November 12, 1987 if the reactor had recently been shutdow While no reliable calculations of reactor coolant heat up rate for PNPS under these conditions are available, it is possible to make reasonable estimates. Past operating experience regarding loss of RHR at other BWRs has shown a reactor coolant heatup rate ranging from 2 F per hour to 22 F per hour under various conditions of reactor decay heat rate and initial reactor coolant temperature. The licensee has estimated a maximum heat up of about 60 F per hour for a recently shutdown plant in cold shutdown condition A loss of shutdown cooling procedure (2.4.25) was referred to early during the event of November 12, 1987. The procedure directs the operator to determine the cause and "take steps to provide a heat sink for decay heat". With regard to vital power the operator is alerted to check 4.16 kV Engineered Safety Feature (ESF) buses and 120v RPS-supplied power to PCIS logic. Additional instruction on loss of one 4.16 kV ESF bus is not provide However, procedures 5.3.18 and 5.3.19 provide some applicable guidance for loss of safeguard buses Y3 and Y4, respectively. In parti- , cular they state that a temporary modification to the high pressure iso- ! lation logic for shutdown cooling would be required. During the event of November 12, 1987, technicians identified the appropriate circuits to be l jumpered and a temporary procedure was written to initiate that activity l had it been necessary to establish shutdown cooling with only one EDG operating. While it is not known how soon shutdown cooling would have been restored if the reactor decay heat rate was high, it is apparent that RHR could have been re-established in a time much shorter than the 14 hours it took to repower bus A6 had it been necessary. l ! _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ ____ _ _ _ _ - l 6.3 Station Blackout Safety Implications While the station blackout rulemaking has not been completed at this ! time, losses of offsite power and preparedness to cope with such eventualities including loss of all ac power at PNPS have been considered i in this inspectio The purpose was to assess the PNPS capabilities regarding station blackout, both in place and planned, and to determine any generic implications for the ongoing station blackout rulemaking activit In summary, the proposed station blackout regulatory requirements include: (1) capability of coping with a station blackout of duration dependent on offsite power and emergency ac power reliability; (2) diesel generator monitoring, maintenance, and modifications as necessary to meet certain reliability targets; and (3) procedures which cover actions necessary to assure reactor core cooling during extended blackouts and procedures to effectuate timely restoration of ac powe The PNPS has experience 19 events involving loss of power from 345 kV lines in 16 years prior to the event of November 12, 1987. There have also been 3 instances where power from 23 kV lines was lost simultaneously with 345 kV sources during that period. The three total losses of offsite power had outage durations of 2 hr/40 min, 8 hr/54 min,* and 3 min. The gives an occurrence frequency for LOOP at PNPS of 0.187 if the November 12, 1987 event is not counted as a total loss of . offsite power and 0.25 if it is. The national average for nuclear power ! plant sites is about 0.08. Because the PNPS losses are primarily due to wind, snow, and possibly salt spray from the ocean, the losses at PNPS are typically longer than the national average of about 30 minute Technical findings developed through NRC analysis in support of the .' prepared station blackout rule show that longer duration losses of offsite power will most likely result from adverse weather and plants located in areas where severe weather can cause a LOOP with relatively high frequency may have a high station blackout ris Durinf, the period 1983-1986 the PNPS diesel generators have had an average unavailability of about 0.02 This value is near the industry average of about 0.02 and corresponds with the higher of two diesel generator reliability levels identified in the proposed station blackout regulatory requirements. BECo has been participating in a pilot safety system unavailability monitoring program which has included the emergency ac power syste The program includes such activities as tracking and trending both equipment and system reliability performance and comparison to targets, assessment of major unavailability contributors, and proposed corrective actions to improve unavailability performance to meet target The program is in the demonstration and assessment stage of implementation and only limited information was made available for inspection at this tim * As reported in NSAC/III and NUREG/CR-3992 l _ ____________________-_ a - _ _ _ 64 Another aspect of PNPS station blackout capability that was reviewed involves procedures for maintaining core cooling during extended blackouts and procedures for restoring ac power. Since the event of November 12, 1987 occurred when the reactor was in cold shutdown, the emergency operating procedures would have been of. limited value if a loss of all ac power had occurred. The emergency operating procedures (EOP's) do address steps to be' taken for a loss of all ac power when the reactor is at operating conditions (steaming). These involve the use of HPCI and RCIC - to maintain core cooling and directing recovery of electrical powe In j fact, procedure 2.4.16 was used both initially and later in a temporary j procedure during the event of November 12, 198 Procedure 2.4.16 is principally directed toward breaker alignments for events involving loss of. offsite power, LOOP with A or B diesel generator failure to start and loss of all ac power (station blackout). s For the latter, it describes de power alignments, cautions the operators on de power capacity and the need to load strip, and briefly discusses the premise that core cooling can be maintained fcr several hours using systems dependent on reactor steam and dc powe In addition, it describes the actions and breaker alignments for restoring power via the startup transformer or "backscuttling" of the unit auxiliary transformer, and.provides several procedures relevant to diesel generator operations and malfunctions. There are numerous other procedures referenced which operators are directed to when conditions warran It is not apparent that past operating experience, especially lessons learned from previous snow and wind induced LOOP, have been factored into the offsite power restoration procedures i n 2. 3.1 In fact, temporary procedure 87-252 was written to deal with the conditions on November 12, 1987. An integrated strategy for coping with a station blackout is also not apparen This observation is based on the fragmentation of proce-dures, potentially many, which could be called upon to successfully cope with an extended station blackout. In these procedures conservation and possible replenishment of dc power and cooling water supplies are only i alluded to. Actions to limit adverse environmental conditions (regarding habitability and equipment operability) are also minima Plant response and the coordination of actions in a timely manner to optimize plant , coping were not apparent. This is not to say that the current plant l capability and procedures are wholly inadequate, but they do warrant up-grading. Discussions with watch engineers indicate that based on their ' experience and plant knowledge, they are prepared to take prudent actions, without spending substantial time sifting through and interpreting the interrelated procedures which would cover the full spectrum of operator i actions during a station blackou As a remedy at least in part, to the procedural shortcoming identified above, PNPS has embarked on a program to develop detailed, integrated, plant specific station blackout procedure. While a draft version of these procedures was not made available for review, portions of a draft study which provides an assessment of PNPS response and capability during an extended station blackout were obtained and reviewed. The scope of the report appears to address the technical issues of concern raised by the i ! ; _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ station- blackout rulemaking and technical findings. The study covers decay heat removal systems, instrumentation and controls, room heatup, de power, and modifications to enhance station blackout capability to cope with a station blackout of eight hours or longe One design modification activity of note which was nearing completion at the time of the November 12, 1987 LOOP, involves the installation of a third - diesel generator as part of the PNPS safety enhancement program (SEP). This non-safety' related diesel generator, rated at 2000 kw, will provide a non-safety source of onsite ac power to the 4.16 kV safety buses. The unit is fully self contained, not dependent on any permanent plant systems for emergency operations, has an independent fuel tank, and a cooling radiato The unit is skid ' mounted and housed in a pre-engineered enclosure to protect it from the environmen It can be made available manually from the cen ral room within one hour of a loss of offsite powe Although a -detailed review was not performed by the AIT, the safety enhancement program (SEP) diesel generator appears to be - capable of meeting the alternate ac power criteria as currently drafted in the proposed station blackout regulatory requirements. It has the potential to reduce station blackout risks at PNPS substantially, the staff has reviewed this installation as part of its review of the PNPS SE The staff review is contained in an August 21, 1987 letter from S. Varga to R. Bir .4 Conclusions and Recommendations Conclusions The loss of offsite power event and other malfunctions which occurred at PNPS on November 12, 1987 did not represent a significant risk at any time during the inciden This finding is primarily based on the very low reactor decay heat levels which existed at the tim Cautious power restoration actions were prudent under the circumstances in existence on l November 12, 198 At higher decay heat levels, equipment malfunctions which occurred would not likely have resulted in a more serious event although personnel actions and procedural adequacy would have been more important. The need i to write Tempora ry Procedures to restore shutdown cooling could have i delayed re-establishing RHR after the "B" diesel generator was secured and ' PCIS was initiated due to loss of power on bus 1 Loss of offsite power combined with loss or unavailability of decay heat removal systems through modes other than loss of onsite ac power sources may also be important at PNP The long time taken to restore offsite power probably could have been shortened considerably but many hours would be required to establish a reliable source of offsite power. The impact of the weather conditions on the availability of the 23 kV power source, had it been operable, is unknown, but previous operating experience shows both 345 kV and 23 kV power supplies affected during winter storm Had the 23 kV source been available and unaffected by the weather, the significance of the November 12, 1987 event would be further reduced. _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - _ _ - _ _ - . _ l ' Relatise frequency and long duration losses of offsite power at PNPS due to severe weather conditions (snow / ice, wind, salt spray) are consistent with the technical findings which support the station blackout rulemakin ! Plant capability and procedures in effect on November 12, 1987 appear to need improvement to satisfy the draft requirements of the proposed station blackout rule. Ongoing efforts in the safety enhancement program which involve assessment of plant capability and development of plant specific station blackout procedures appear to meet the intent, at least in part, of the proposed station blackout requirement Installation of the SEP diesel generator has the potential to sub-stantially reduce station blackout risk This wi l,1 be especially important if switchyard improvements prove ineffective in eliminating long duration total losses of of fsite power in the futur Recommendations The loss of shutdown cooling procedure should be revised to address loss of power to either safeguards panel Y3 or Y4 both with and without offsite power availabl Procedures for restoring offsite power should be reviewed against past operating experience, especially events caused by severe weather, and revised to reflect lessons learned and anticipated problems which may need resolution to optimize power recovery time. l . _ _ _ _ _ - _ _ _ _ _ _ ., ! APPENDIX A ENTRANCE INTERVIEW ATTENDEES NOVEMBER 16, 1987 Boston Edison Company (BECol H. Balfour, Training Group Leader R. Bird, Senior Vice-President, Nuclear M. Brosee, Maintenance Section Manager G. Edgar, N&H/BECo P. Farron, Staff Assistant J. Flannagan, Consultant R. Grazio, Field Engineering Section Manager P. Hamilton, Compliance Group Leader J. Howard, Vice-President, Nuclear Engineering and Quality Assurance S. Hudson, Operations Section Manager J. Jens, Radiological Section Manager R. Ledgett, Executive Assistant to Senior Vice-President Nuclear A. Morisi, Assistant Director of Outage Management J. Pawlak, Power Systems Gr,up Leader .V. Peters, Consultant K. Roberts, Nuclear Operations Department Manager E. Robinson, Nuclear Information Manager J. Seery, Technical Section Manager R. Swanson, Nuclear Engineering Manager B. Tucker, Senior Engineer E. Ziemianski, Training Section Manager ! U.S. Nuclear Regulatory Commission (USNRC) T. Kim, Resident Inspector Pilgrim J. Lyash, Resident Inspector Pilgrim C. Warren, Senior Resident Inspector Pilgrim _ ______ -__ _____ _- _ - APPENDIX B ' EXIT INTERVIEW ATTENDEES LIST NOVEMBER 20, 1987 Boston Edison Company (BECol R. Bird, Senior Vice President Nuclear W. Clancy, Systems Group Leader R. Deacy, Security Operations Group Leader R. Fairbanks, Design Section Manager P. Farron, Staff Assistant R. Grazio, Field Engineering Section Manager P. Hamilton, Cenpliance Group Leader S. Hudson, Operations Section Manager J. Jens, Radiological Section Manager R. Ledgett, Executive Assistant to Senior Vice-President Nuclear A. Lee, Onsite Emergency Preparedness Coordinator, EG&G B. Lunn, Senior Compliance Engineer P. Mastrangelo, Chief Operating Engineer J. Mattia, Quality Assurance Group Leader A. Morisi, Assistant Director of Outage Management K. Roberts, Nuclear Operations Department Manager E. Robinson, Nuclear Information Manager J. Seery, Technical Section Manager R. Sherry, Maintenance Group Chief Engineer A. Shiever, Operator Training C. Stevenson, Senior Compliance Engineer R. Swanson, Nuclear Engineering Manager B. Tucker, Senior Engineer U.S. Nuclear Regulatory Commission (USNRC) T. Kim, Resident Inspector Pilgrim J. Lyash, Resident Inspector Pilgrim C. Warren, Senior Resident Inspector Pilgrim i l -_____ -_ _ _ _ _ _ _ _ _ - _ - APPENDIX C PERSONS INTERVIEWED NOVEMBER 17-19, 1987 The below listed individuals were formally interviewed by the NRC team during the course of the inspection. Other personnel were contacted as the inspection interfaced with their area. R. Atkins, Senior Electrical Engineer R. Bird, Senior Vice-President Nuclear N. Brosee, Maintenance Section Manager J. Farrel, Electrical Lab Engineer J. Giar, Nuclear Watch Engineer B. Higgins, Electrical Lab Engineer J. Howard, Vice-President, Nuclear Engineering and Quality Assurance J. Jens, Radiological Section Manager G. Lafond, Staff Engineer, Maintenance R. Ledgett, Executive ~ Assistant to Senior Vice-President Nuclear B. Lyons, Nuclear Operations Supervisor M. Maguire, Senior Electrical Engineer P. Mastrangelo, Chief Operating Engineer J. McCann, Nuclear Watch Engineer S. Minahan, Nuclear Watch Engineer P. Moraites, Assistant Chief Maintenance Engineer K. O'Donnell, Staff Engineer, Maintenance W. Olsen, Nuclear Watch Engineer J. Pawlak, Power Systems Group Leader J. Purkis, Senior System Specialist K. Roberts, Nuclear Operations Department Manager J. Seery, Technical Section Manager S. Shidner, Electrical Supervisor J. Smith, Electrica Lab Engineer P. Smith, Technical Group Chief Engineer R. Swanson, Nuclear Engineering Manager K. Taylor, Nuclear Watch Engineer G. Van Epps, Nuclear Operations Supervisor D. Williams, Nuclear Watch Engineer _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ - _ _ - _ _ _ - . _ _ -__ - -_ ] j APPENDIX D INTEGRATED SEQUENCE OF EVENTS i The following sequence of events was assembled from various plant logs and personnel interviews, as noted. There are obvious variations in the times recorded in different sources for the same event. Where this was crucial * ' to the understanding of the event, the most reliable source was used in the event analysis by the tea Where log entries were not clear or brief statements, they have been supplemented with commentary for clarit SEQUENCE OF EVENTS PILGRIM NUCLEAR POWER STATION NOVEMBER 12, 1987 I LOSS OF OFFSITE POWER EVENTS ; November 12, 1987 TIME EVENT SOURCE 1:27 kV Line No. 342 Computer Alarm degraded voltage Log 1:27:05 Computer analog altrm received indicating 338.73 k Alarm setpoint is 341.50 k Alarm never clears indicating that voltage remains below reset value until the Loss of Offsite Power (LOOP). Operators not aware of alarm Operator and therefore did not enter Interviews Degraded Voltage Procedure (2.4.144) or monitor 345 kV line/ ring bus voltages to assess stability of 345 kV grid, 2:05 Loss of Offsite Power Relay operations for lines 342 and 345 kV line No. 342 355, 2:05:26 "A" to "C" to ground fault, ACB 103 and 104 open, ACB 104 was slow in opening, resulting in a transfer trip of breakers 105, 2130, and 312. I D-1 _ _ _ - _ _ _ -- . _ _ _ _ _ _ - - _ . _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ - _ - _ _ _ _ - . _ _ _ _ - . __ _ - _ _ _ _ _ - _ . . Appendix D TIME EVENT SOURCE 2:05 a.m. (con't) 345 kV line No. 355 "B" to "C" 2:05:34 phase fault ACB 1670 open Startup transformer differential leads'to ACB 102 openin 'kV line No. 355, breaker 1670 reclosed and reopened four more times, 2:06:56, 2:13:05, 2:15:01 and 2:15:55. REMVEC ordered the breaker manually opene :06 kV Yard NOS Log, ACBs 102, 103, 104 Operator and 105 open Interviews Full reactor scram Computer : due to loss of power Alarm Lo to RPS MG sets which 2:09:56 - are fed from 480 Vac 2:11:31 MCCs B22 and B23 Full PCIS due to NOS Log loss of power to 120 Vac buses Y3 and Y4 which are fed from 480 Vac MCCs B17 and B18 as well as de-energized RPS abov This isolates RHR SDC (Group III isolation) Full RBIS due to PCIS NOS Log EDG's "A" and "B" start NOS Log due to undervoltage on 4.16 kV Buses A5 and A6, EDG's tie on to Buses A5 and A Running air compressor K-111 stops due to loss of power to 480V, MCC B5 which results in depressurization of instrument air syste :12 NWE and electrical supervisor NOS LOG, respo.7d to switchyard relay house Elect. Supervisor to investigate event 0-2 - _ _ _ _ _ _ - _ _ _ - _- __ - l Appendix D TIME EVENT -SOURCE 2:20 Control room notified Nuclear NOS Log Operations Department Manage REMVEC ordered opening of T-900 REMVEC Switching disconnect on line No. 342 in Order switchyar :23 Re-energize RPS Bus "A" from MCC B-10 . When RPS Bus "A" is Computer re-energized, all "A" trips Alarm Log clear which allows reset of 2:22:55 a.m. - PCIS and restoration of RHR 2:23:44 ' in SDC mode. At this point neither "A" nor "B" scram has Operator been rese Interview ' Reset PCIS and RHR Isolation Logged at 0221, but occurred after NOS Log ; RPS Bus "A" re-energized abov ' This must be done before attempt- .i ing to re-open RHR valves 47 ' and 50 (RHR pump suction valves). 2:24 a.m. Started CRD Pump "A" NOS Log 2:27 Started RWCU Pump "B" NOS Log 2:30 Attempted Start of RHR Pump "A" NOS Log Received PCIS/RHR Isolation Operator Interviews Cause unknown 2:31 RPS MG Set "B" Reported to NOS Log be running on normal supply Operator i at reduced Voltage and Interviews Current NOS felt that "B" EDG might be overloaded and directed that 480 Vac Bus B4 be stripped and the feeder to B4 from 480 Vac bus B2 be opene D-3 _ _ _ _ _ _ - _ _ - _ _ - . - - _ --- -- - ---_--- . _ _ _ _ _ _ _ _ _ . , . Appendix D ! l TIME EVENT SOURCE 2:31 a.m. (con't) EDG "B" was loaded to 1300 K ! (Rated load 2,600 KW). ) "B" MG set was probably coasting down from the initial loss of powe ! 2:35 Shitted 120 VAC RPS NOS Log Bus "B" from "B" MG Operator set to 480 VAC MCC Interviews B10. De-energized 120 VAC RPS Bus "A". MCC B10 can only power one . RPS bus at a tim ! This caused a "B" scram signal Computer and allowed clearance of Alarm Log "A" scram trip contacts 2:36:57 a.m. - in RPS/PCI :38:55 Notified Maintenance Section Mg AD00M Log 2:36 Reset PCIS/RHR Isolation NOS Log Reopened RHR valves 47 and 50 Operator and re-established SD Interviews Moderator temperature slowly decreasing without pump heat from RH :45 Started RHR Pumps "A" and "C" NOS Log 0240, 0245 RHR Loop "A" flow analog Computer Alarm Log computer alarm clears 2:45:04 indicating 3267 GPM (one pump running). Subsequent pump start would not be indicated since flow already above alarm reset poin :46 Stopped CRD Pump "A" NOS Log 2:55 Notified NRC - Headquarters NOS Log Operations Officer (H00). ! D-4 { l i _ _ - - - _ - - - _ _ _ _ _ _ _ _ - - _ - _ _ - _ _ _ _ _ _ - ,__ - ______ _- Appendix 0 TIME EVENT SOURCE 3:00 NOM arrived on-site NOS Log 3:10 Air compressor K-104B placed NOS Log into service, essential System Specialist instrument air header repressurized Interview to 40-50 psig; insufficient pressure to reset. scram discharge header vent and drain valve :15 Started RWCU Pump "A" NOS Log 4:00 Nuclear Operations Department Management Manager (NOM) notified Senior Interview V.P.- Nuclear and NRC Resident Inspector of even :05 % of 345 kV switchyard Sr. Elect. Engineer insulators covered with snow Interview from top to botto :30 - 5:00 a.m. NOM notified Nuclear Information Management . Manager and Middleboro State Interview ! Police Barrecks (local Civil Defense) of even NOM attempted to notify (Chief Operating Engineer), but phone inoperabl :00 Decision made to wash switch- Sr. Elect. Engineer yard insulators in preparation to Interview ; ' return startup transformer to servic ! 5:07 ' Stopped RHR Pumps "A" and Computer Alarm Log "C" and RWCU Pump "A" 5:07:05 In preparation for re-alignment of 4.16 kV NOS Log electrical system in accordance with Procedure 2.4.16 5:09 De-energized 120 VAC RPS Computer Alarm Log Bus "B" and energized 5:09:21, 5:09:27 VAC RPS "A" from MCC B10 NOS Log 5:10 D-5 ) Appendix 0 TIME EVENT SOURCE 5:17 Energized 120 VAC RPS Bus "B" Computer Alar, Log from B22 5:17:14 NOS Log Fed from B2, B4, B22 through RPS MG set "B" 5:18 Reset Scram Computer Alarm Log 5:18:55 - 5:18:57 NOS log indicates reset 1/2 NOS Log scram, but computer log indicates 0515 full rese Reset PCIS/RHR Isolation NOS Log 0516 , , Started RWCU Pump "A" NOS Log 0517 5:30 a.m. Started RHR Pumps "A" and "C" Computer Alarm Log-5:30:04 ; Analog flow for RHR loop "A" NOS Log is 6880 GPM indicating two pumps runnin :55 a.m. Blown fuses reported in NOS Log Analog Trip System (ATS) Panel Operator Interviews 6:00 a.m. Maintenance personnel ready to ' start washing insulators in Interview 345 kV switchyard to restore Sr. Elect. Engineer No. 355 line and startup transformer. 6:29 Blown Fuses in ATS Panel Computer Alarm Log replaced 6:29:27 MR 87-726 6:30 NOM notified Executive Management Assistant of even Interview 6:57 NWE and REMVEC approve washing Procedure of insulators in 345 kV switchyard. 3.M.3-20 7:30 Mod Management Group to determine AD00M Log outstanding work which prevents using 23 kV shutdown transformers. 7:45 Stopped RHR Pump "C" NOS Log D-6 r__ _ _ _ _ _ _ . . . [.' L Appendix D.- TIME EVENT SOURCE 8:00 "A" priority' maintenance Elect. Log Request to troubleshoot air Maint. Request compressor K-104-C for motor F&MR ground, reference MR-87-72 :05 Started RHR Pump "B" in NOS. Log Suppression Pool Cocling Mode (SPC)- 8:55 REMVEC ordered T-901 disconnect REMVEC switchin opened on line No. 355 order 9:02 kV line No. 355 re-energized Sr. Elect. Engineer with switchyard disconnect Interview open, line tested goo :30 Senior Electrical Engineer AD00M Log ! and staff have checked startup ' transformer and found no problems; they are prepared'to wash insulators and re-energize from normal 345 kV source (estimated time 2 hrs.) i 9:40 REMVEC ordered disconnects 104A REMVEC Switching and B opene Order 10:00 I&C Supervisor discovered "C" Operator phase ammeter on "B" EDG Interview Breaker ( A-609) reading zer :45 Electrical Lab checking "B" diesel NOS Log generator, no indication on "C" phas :00 Line No. 342 was made available to Sr. Elect. Engineer the Pilgrim statio Interview 11:20 Notified to secure "B" diesel due NOS Log to "C" phase proble :23 Stopped RHR Pump "B" Computer Alarm Log SPC secured 11:23:04 Performed in preparation for NOS Log stopping "B" EC :20 D-7 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - Appendix 0 ; i TIME -EVENT' SOURCE 11:25 RBCCW Cross-tied NOS Log Performed in preparation for i stopping "B" EDG Started unloading EDG "B" NOS Log 11:30 "B" diesel generator unloaded NOS Log and breaker tripped; reclosed automatically on the bu NED wants testing of startup Sr. Elect. Engineer transformer, Electrical Engineering Interview ' was requested to initiate backfeed through the auxiliary transforme Air compressor K-1048 stops due Operator and I to loss of power which results in System Specialist depressurization of essential Interviews instrument air heade '11:31 Reactor Scram Computer Alarm Log 11:31:36 Due to loss of A6 bus when EDG "B" secure :35 "B" diesel generator stopped NOS Log using key lock switch, received full scram, lost shutdown cooling, TBCCW and instrument ai I 11:55 and 250 voit de batteries are ' NOS Log placed on backup charge :35 Mode switch placed in shutdown NOS Log I mod :08 TSC partially staffe TSC Log 1:20 REMVEC authorized opening of T-931, REMVEC Switching i T-930, and 1018 disconnects and Order i closing of T-900 and T-901 disconnect :30 Doble crew ordered from Watertown Sr. Elect. Engineer to Plymouth to test startup Interview transforme D-8 - _ _ _ _ _ - _ _ _ _ _ Appendix D TIME EVENT SOURCE 2:00 Parallel path activities in AD00M Log progress: (1) a diesel driven, oil free air compressor is expected to arrive onsite at 5:00 p.m.. The compressor will be connected through a previously approved modification; (2) Plan ! to Hi Pot the high side of the startup transformer, megger the low side and sample the oil for gases; (3) electricians are working toward "backscuttling" to feed through the main and auxiliary transformer :15 Lines No. 355 and No. 342 were Sr. Elect. Engineer energized through Pilgrim Interview switchyard with startup transformer, and breakers 104 and 105 isolate Power restored to lines No. 342 NOS Log and No. 355 by closing ACB's 102 and 10 Disconnect between startup RI Log transformer and bus opene :20 :45 Health Physics reported R.P. Shift Log Radwaste elevation (-1 ft) flooded, water backing up through drains from turbine building sum :47 Control room requesting fire TSC Log protection engineer to assess condition of plant deluge systems (loss of air), need to isolate and post fire watches 3:00 Tagging completed for backfeeding Sr. Elect. Engineer through auxiliary transforme Interview Start isophase link remova l l D-9 . . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - ____ . - - _ - _ - - _ _ Appendix 0 ' TIME EVENT- SOURCE 3:50 - 4:10 Fire truck moved to tie-in TSC Log point, hoses laid out, ready for backup. (Note: This was proposed, but never occurred.) 4i10 Main generator isophase bus AD00M Log links remove :35 Notified problem with current AD00M Log transformer on "C" phase of "B" diesel generator was a burnt lead on the Watthour mete Electricians will replace lu :00 Isophase links removed; Doble Sr. Elect. Engineer crew arrived at station; "B" Interview diesel generator current trans-former lug repaired; meggering test performed on secondary of startup transformer and determined acceptable. Washing of insulators started to support backfeed through auxiliary transformer, i 6:00 Startup transformer gas test Sr. Elect. Engineer { results show transformer is Interview { acceptable; transformer oil I sample taken for analysi :15 Notified that red tags on AD00M Log MR 87-46-190 need to be removed to complete "back scuttling" on auxiliary transforme I 6:33 Hi Pot tester being setup to TSC Log l test startup transforme j { 6:57 Doors between diesel generator TSC Log l rooms closed; ventilation lineup l was lowering "B" diesel generator { I room temperatur :00 - 8:00 Doble crew onsite to test Elect. Log startup transformer. Electricians removing cables from bushings ) in preparation of tes I l D-10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - - _ _ _ _ _ _ _ . ._ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ - _ _ _ - _ _ _ _ - _ _ . _ - _ _ _ _ _ _ _ _ - _ _ _ . Appendix 0 TIME EVENT SOURCE 7:00 - 800 Secondary of startup transformer (con't) meggered, 100 megohms all three phase :00 Insulator washing completed for Sr. Elect. Engineer backfeed through auxiliary Interview trans forme :30 Received permission to remove AD00M Log tags on MR 87-46-190. (Note: see 6:15 p.m. entry) The Maintenance Section Manager wanted the person responsible for the tags to personally notify the watch engineer that tags can be remove :30 p.r Watch engineer received permission AD00M Log to lift red tags on MR 87-46-190 to allow "backscuttling." 10:00 Oil sample on startup transformer Sr. Elect. Engineer tested good per telephone Interview conversatio :50 "Backscuttle" failed because AD00M Log disconnect pin was stuc Maintenance to disconnect-solenoid for pi :58 Disconnect linkage freed, TSC Log disconnect close :09 Closed ACB 105 energizing the NOS Log auxiliary transforme :15 Energized Buses A6 and B1 through NOS Log auxiliary transformer backfee Energized A-4 bus AD00M LOG 11:25 Energized A-3 bus AD00M LOG 11:45 Air compressor K-111 energized NOS Log 11:49 Pressurizing the instrument NOS Log air syste D-11 _ __ __ _ _ - ___ _ __-_ _ _ _ ____ _ _ _ _____ - _ - l ' Appendix D . TIME EVENT SOURCE 11:55' Energized B-4 bu AD00M LOG . November 13, 1987 l 12:05 Air system attains normal Operator.and 102-108 Psi System Specialist Interview 1:00 a.m. Reset Scram NOS Log, 1:10 a.m. Started RWCU Pump "B" NOS Log 1:20 a.m. Startup transformer work AD00M Log complete :50 a.m. Started RHR Pump "A" NOS Log Restored RHR in SDC mod :55 a.m. Identified problem with AD00M Log "B" EDG Lube Oil Prelube Pump Should have started automatically on restoration of power to MCC B1 :00 a.m. Reset RBIS NOS Log Testing completed on startup Sr. Elect. Engineer transforme Interview , 3:15 a.m. Took reciosing of ACB 102 and NOS Log ACB 103 per REMVEC instructions REMVEC Switching opened'ACB 103 and 10 Order 3:30 Closure of T-931 disconnect REMVEC Switching authorize Order 3:35 a.m. Closed in ACB 102 and 103 and put NOS Log reclosing to "on". Energized startup transforme :30 Preaction Fire System secured NOS Log I to bring in temporary heater to warm "B" EDG D-12 h : F ~ > ,,, , J. Iri Appendix D TIME EVENT SOURCE 5:30 Startup transformer energized S lect. Engineer and ready for' servic Interview * ' 5:40 a.m. ACB tagging status: Elect. Log ACB 104 A&B disconnects open and j tagged ACB stuck breaker - off, no ta MR-87-731 meggering and Doble testing of the startup transformer complete .) Transformer is available for service if neede :00 a.m. EDG "B" Lube Oil Prelube Pump F&MR 87-637 troubleshoot complete Pump drew locked rotor current and tripped when starte :40 a.m. Report that the "B" diesel generator AD00M Log prelubrication pump will be changed , .i 12:23 p.m. EDG Lube 011 Prelube Pump MR 87-737 replaced, returned to service TSC Log 1:50 p.m. Diesel generator pre-lubrication TSC Log pump is installed and being tested 2:40 p.m. Pre-lubrication pump on the "B" AD00M Log diesel is installed but there is some concern regarding the direction of rotatio l 5:50 p.m. MR-87-737 is complete, diesel pre- AD00M Log lubrication pump running satis-factorily, oil was warmin j 8:03 p.m. Started "B" EDG for NOS Log surveillance Monthly manual start and load per procedure 8. :30 p.m. A priority "A" MR is being written AD00M Log for the "B" diesel generator for leaking injector D-13 - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -_ _ _ _ _ . _ - _ _ _ _ _ _ . _ - .-. > Appendix D TIME EVENT SOURCE 10:00 Stopped EDG "B" NOS Log TSC Log Due to injector leaks November 14, 1987 12:15 MR issued on "B" diesel generator, NOS Log tagging out "B" diese :35 Started to energize Al and A2. buses NOS Log via the startup transformer. . Buses A3 and A4 on the startup transformer. I 1:45 Started Comdensate Pump "B" AD00M Log 2:05 Started /SSW Pump "B" AD00M Log NOS Log 2:07 Bus A6 transferred to the startup NOS Log transformer. l 2:45 Stopped RHR Pump "A" and RWCU NOS Log l- ' Pump "B" TSC Log 2:55 .16 kV Bus A5 transferred to TSC Log S/U transformer AD00M Log 3:00 Started "B" RWCU Pump TSC Log NWE Log 3:30 "A" diesel generator off the line, NOS Log 3:35 Reset RBIS NOS Log j 3:45 Secured TSC suppor TSC Log 4:35 Opened ACB 104 & 105, secured AD00M Log backfeed 4:45 RBCCW LOOPS split NOS Log 5:30 Closed ACB 104 AD00M Log 2:50 Started EDG "B" for NOS Log ; Surveillance f-f .. D-14 't -. -_-___ -__ Appendix D q , 3:16 p.m. Stopped EDG "B" due to Problem NOS Log with Bolting on Fuel Rac :51 p.m. Started EDG "B" for Surveillance NOS Log 4:15 p.m. Stopped "B" EDG NOS Log 5:00 p.m. Restored "B" EDG Preaction BECo Chronology Fire Syste i 5:08 p.m. Started EDG "B" for Surveillance NOS Log ; a 7:55 p.m. Stopped "B" EDG NOS Log 11:15 p.m. EDG "B" Declared Operable BECo Chronology l l l l , l l i D-15 l m-. _ my . - _ _ _ _ _ _ _ _ _ _ _ _ _ .- _ _ . . _ _ _ _ _ _ . _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _______ _ _ _ _ _ _ . #k"MAPPENDIX'E-1 BECO REC 0VERY SCHEDULE , _ . . . g . . _ . . - - _ . g... . . . . . k m- >oo . e - - - . - "Z e< - N - 9 3' D 4 . ed . k , - .. p 61J c3 E l: g a F ._ m C s= E / > [ . .k g,a .) g~ E Mk W 5 -% e .; b . * S- - O d 4 j - a .. . . 1 - * 7* 1 o h ... ng _ . . . _ . .. _ . . . w . . . _ . . . . . . - _ . 5' .. . g . . . . . . . - - n . .. .._ . . _ . . . . _ . _ . . . . _ .._ _2 -_ . - . _ . . . . . _ . . . . . . _ . _ . .b . . _w E g- .- , - . - .- * . . . r r (. l- . . _ . . _ .. - et -- .._ _ .._... .. -- - - a - --- . %, , 2 ,--- - - - - - - - - - ~ - - ~ - - ' . . _ . . . . . . . _ _ . . _ _ . . _.f . '. _ . . . . _ ___L% . . . .. - . _ _. -....-. . . . . ,g . . - . 4. _. . , =_ _ . . . .. y .. . - _ ao . - .. . -.2 A .- .*- r .. . ..c o - . . ** . . 2 ' I o . Y - - .. . . . . _ . . _ . . . . . . _ - _ _ . . - - . . _ -. . - . . . _ - . f, g-- _,, . d w .,f a . _ _ . _ _ .x,_____ .. ; .}_ ._ __ _ _ . . __ ___g .g. _gg _ . - _._p_g $ .__. __.,_ g_).__ # -. -_ . .-.] ._ . c' .v ul _ g"..._k __..; o*_ 3 _ . . . _ . [ cd v1 . . . 2, - ' .. d .O M _ _ . . . - _ . . _ _ . . . . . . . l f*1 e - . .d 3 y 2 f ..d _3 . . _ _ . g _ _._. ,.._'. __., .____-..w . _3 . ._ _ . _ _ . ._ . _ c . ,I . .- ..f3 _ . .- - . - # x .__._., - . - - . . _ . . . . .. 32 . _ ~ . . .. . . # . . .. s _ . . . ._ . . . _g .. , . ..- o ._ . +.n-p x . ._. -. . .x . . .._ .. . .._ . *0 . . . _ _ _ : _ .. _. _. _ g: . _ . . - . - . . _ . . . -_ . . _ t E ,8 _ _._e. g _.:Y_g _ . '. __g - ...Q-._.____._.__., - . s i .. -. , , . . _ . . . . . _ . _ . . _ . . - .._ . _ . _ . . . _ . _ _ _ _ _.. __ . ' _I~-d - _ - . _ _ _ . . . _ . - . . - . . _ _ _ _ _ _ . _ - - . . . - M _ _ _ _ R_q', > . . _ _ _ _ _ _ . . _- . w ._ . ' ___ LA _ _ _ _ _ _ _ _ _ _ _ _ _ . __ - 4 .__ _ . s , i _ - . . _ .-._m..---._..-.__--.~.- -. J _ _ - _ _ . _ " APPENDIX E-2 BECO RECOVERY SCHEDULE h\Ne {- , ' - ' .o C #_ >C H B 9.NJl4 c M_ e @ 3 zw> r e ca _ g a - I-l- .s E $ si 'ogy'il E > - - b / b .f a-a : - % h cm s = C4 N y * U E I . g- < A - - .g % <t m Q ~ k4 a <#9 , e o ea~ow gd R g~ ~ s Ef * 'wna w Q f- Ef gs wE g- n a s G v a N . -> p g ~ - n g5 $ ; g_ a 5 / e _ .__0 ;.,g?) o, x i t-g _-. c n a.__.....___.._y" . .- . .. ,._ _ I g . g_ g q .q , _ _ _ . __ . C .__) c .- _._ _ - __ J n .. _ .. o l._g'o 2 a 3 e j_ ..'+ %q U j 45 ' re - L, , 3 e ,u@ am 1 =d M k d re U " m - S T Q &_ 2 p ._ a g . y -. _ . ha ? r g:; a - t-e ... "8 0', ' r (* mh2 " E y ~y m ' ap o- E r e- b2 * u N,; Hs p { ' & l* y ; 3- 'in - *e F o 00s<o-e B e er t - S a S E ,3 p s a< g M 0 *E b; a g ~ -- - _ ____-______-__________-_________________________________ - APPEf1 DIX E-3 Cge ) BEC0 RECOVERY SCHEDULE O g9d $ i F 9 cg s !! ! = e %a 3&A, i bg g . > * b . ; e c ; . . y 1 4 F s a 8-s et - - - - - - -? 'g g l Ws $ R- a.- L p p .. g k 5 a e g_ c Og l g i I s i gE E l- E a, l E p, i l". g F lls d -, !a," l<> gm I e p_ e #e -a rl%._-g-i-___ i i bgg-* -- - lh g ~~i V " l - o a a i u ap' B o I I i - i T e s * 3 h- S ,o * IJ " ! " E 62 hh h----E -$ .<> , V $L$ V f ?a' i, F # - bo d HI . ! g~' it -L.4p t a 8, m s e E < s2 e A w a = c = '* ^ .- _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ . _ _ _ _ _ _ _ - _ I APPENDIX F INSTRUMENT AIR SYSTEM INTERRELATIONSHIPS High Pressure Air System A loss of high pressure air affects the following system * Clean radwaste system - seal air is lost to the flat bed filters, F-5 * Reactor water cleanup (RWCU) system - blowdown air for the clean-up filter demineralized is los * Condensate demineralized system - sluiting air is lost * Solid recovery system - air to the spent resin transfer pump and the solid recycle recovery pump is lost * Screen wash system - air to the intake structure screen wash bubbler is lost Non-Essential Instrument Air A loss of non-essential instrument air affects the following systems: * Resin cleaning system - air operated ( AO) valves in the resin cleaning system will shut, except the connection between the cement feeders to the mixer / feeder which will fail "as-is"; air to the dust collector, which supplies air to the cement bin, is lost * Clean radwaste system - clean radwaste flatbed filters and instrument seal air is lost; level indication for the radwaste tanks is lost * Radwaste collection system - turbine building equipment drain sump pump isolation valves fail shut; treated water hold up tank instrumentation and valves are inoperable * Primary cor.tainment - air is lost to the MSIV's, the MSIV accumulators provide reserve air pressure to shut the valves; torus make-up valves fail shut; and, reactor building / torus vacuum breakers will fail open if accumulator pressure is los * Turbine building closed cooling water - Valve A0-4160, cooling water to mechanical vacuum pump seal water cooler, fails ope * Condensate demineralized system - all valves should remain "as-is" except for the sluicing valves which will " fail-shut" * Demineralized water system - demineralized water (DW) transfer valve from the makeup demineralized system to the DW storage tank will " fail-shut" as will the transfer valve between the DW storage tank and the condensate storage tank; city water supply to the filter valves " fail-shut" as will the discharge from the acid pumps; and, level indication will be lost for the caustic storage tank and the acid storage tan F-1 _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ - _ _ _ _ _ - Appendix F * Diesel oil system - diesel oil day tank isolation valves fail open * Fuel pool cooling system - fuel pool make-up valve fails "as-is"; skimmer surge tank level indication is lost Essential Instrument Air When essential instrument air is lost the following major systems are affecte * Compressed air system - the following valves fail in the closed position: A0-4365 - non-essential instrument air isolation A0-4353 - HP/LP service air cross-cor.nect isolation A0-4356 - essential instrument air isolation to containment atmosphere control system A0-4310 - instrument air dryer / filter bypass A0-4350 - HP service air isolation * Main condenser vacuum - the main condenser vacuum will decrease as the steam jet air ejector steam supply valve closes; the main condenser vacuum breakers will fail as is * Feedwater heating - bleeder trip valves fail shut and spill valves fail open thus eliminating all feedwater heating; the level control ; valves fail closed and the high level dump valves fail open, diverting all drainage to the main condenser * Cooling water - reactor building and turbine building cooling water heat exchanger bypass control valves fail closed, providing maximum cooling to the RBCCW and TBCCW cooled components; the RBCCW and TBCCW head tank make-up valves fail open * Off gas system - the off gas system isolation valve to the stack (AO-3751) will fail shut * Control rod drive system - the flow control valves will close, scram valves will open under spring pressure (inserting the control rods) and the instrument volume vent and drain valves will close * Feed and condensate - Feed regulating valves fail "as-is"; the condensate pump recirc flow valve fails open; the feed pump recirc valves fail open; and the main condenser vacuan breakers fail 'as is" * Standby liquid control poison tank level indication is lost F-2 - - _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . . _ _ _ _ _ _ _ _ _ _ . _ _ - _ _ _ . __ -_ Appendix Turbine Building Closed Cooling Water The TBCCW system supplies cooling water to cool the compressed ai It also condenses moisture in the air to aid in moisture removal. Losing TBCCW may cause severe damage to the compressor. If the compressor is started without cooling water, it must be shutdown immediately and allowed to cool down prior to initiating cooling water flo F-3 - _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ . ._ , _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _ . _ _-_ _ ____ __ _ __ _ _ _ _ APPENDIX G SEQUENCE OF EVENTS LOSS OF INSTRUMENT AIR Approximate Time Event Source November 12, 1987 2:06 Loss of offsite power occurs, which NOS Log causes running air compressor K-111 to sto Nuclear operations supervisor (N0S) refers to procedure 5.3.8, Loss of Instrument Ai :20 Air system block valves actuate to Operator and - 2.:25 a.m. isolate essential instrument air header System Engineer from high pressure service and non- Interviews essential instrument air header :35 Essential instrument air header Operator and - 2:45 pressure continues to drop off and System Engineer reaches essentially zero psi Interviews 3:10 Compressor K-104B placed in service; NOS Log essential instrument air header commences System slow repressurization to 40-50 psig; Specialist insufficient pressure to reset scram Interview discharge header vent and drain valves, rated to function at 70-110 psi :00 Systems Group NSSS mechanical lead System senior system specialist arrives Specialist onsit Interview 6:30 Control room operator experiencing AD00M Log problems with station air system because only one compressor available powered off essential power bu :30 Nuclear operations department manager AD00M Log (NOM) directed systems group to investigate availability of clean air compressor for temporary tie-in and to pursue options for restoring air system pressur G-1 l .- _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ _ _ _ _ _ _ _ _ Appendix G 8:00 Priority "A" maintenance request written System l to repair compressor K-104C; determined Specialist no suitable (oil-free) portable air Interview compressors onsit '8:15 Vendor (Atlas-Copco) contacted to locate System compressor Specialist Interview 9:00 Atlas-Copco unit located in Connecticut System Specialist Interview 9:30 Senior system specialist for air systems System-10:00 arrives'onsite; conducts verification Specialist walkdown of air system response functions Interview and status, investigated alternate means of pressurizing header, recommended securing compressor K-104B to depressurize air heade :00 Atlas-Copco driver and transportation System arrange Specialist Interview 11:30 "B" emergency diesel generator is NOS Log tripped manually which causes Operator compressor K-104B to sto Interview 11:50 Essential instrument air header Operator and - 12:00 a.m. pressure drops off to essentially System zero psi Specialist Interview 12:00 Portable compresser unit in transit, System ETA 5:00 Specialist Interview 1:40 Air compressor K-104C is tripping TSC Log on low oil pressure during post-maintenance testing of breaker terminal repair :43 TSC assesses compressor K-104C oil TSC Log ; - 1:51 pressure problem; pressure should be ! 35 psig but is not building up beyond 4 psi G-2 _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ . _ - _ _ _ - _ _ - _ _ _ - Appendix G ' 2:23 TSC engineers dispatched to identify TSC Log outside/onsite available air sources; reported a compressor outside the radwaste truck lock; researching its capability (capacity, oil free?) ' 3:00 Air compressor K-104C tagged out of NOS Log service; was tripping on apparent low oil pressur :24 Control room briefed on fire TSC Log protection air-operated valves and fail positions and accumulator :22 Portable compressor arrived at the TSC Log site gat :30 Compressor K-104C. work continuing Mechanical to replace the oil pum Installing Maintenance Log temporary modification tie-in for Atlas- Copco compresso :07 Installing temporary modification NOS Log 87-30, Temporary Instrument and Service Air Suppl :10 Portable unit onsite.
9:22 Temporary modification (TM) 87-30 and TSC Log - 9:35 associated safety evaluation for installation /use of portable unit is approved by onsite review committee and NO :30 Operations, section completed TM 87-30 Operator and valve lineup, systems group received System direction from control room to keep Specialist portable compressor in standby mod Intervie :09 Offsite supply breaker ACB-105 NOS Log is close :45 Air compressor K-111 re-energize NOS Log 11:49 Pressurizing instrument air syste NOS Log G-3 L____--____ _ _ _ ___ _ .- . _ _ _ _ _ _ _ _ _ . _ - . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ________ Appendix G-November 13, 1987 12:05 Air systems attain normal 102-108 Operator and psig operating pressure ban System Specialist . Interview 1:30 Compressor K-104C tested satisfactorily Maintenance 'following repair of electical problem Request and oil lea :30' Compressor K-104C tags /isolations Maintenance removed and returned to. normal lineu Request 10:40 Compressor K-104C repair is complete TSC Log I i l l G-4 _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ - _ _ - w APPENDIX H > ACRONYMS AND INITIALISMS ACB Air Circuit Breaker AD00M Assistant to Director of Outage Management AIT Augmented Inspection Team ANSI American. National Standards Institute ATS Analog Trip System CR0 Control Rod Drive EAL Emergency Action Level EDG Emergency Diesel Generator F Degrees, Fahrenheit , GPM Gallons per Minute ! kV Kilovolts LOOP Loss of Offsite Power MCC -Motor Control Center M3 Motor' Generator NED Nuclear Engineering Department NOD Nuclear Operation Department NOS Nuclear Operations Supervisor NWE Nuclear Watch Engineer PCIS Primary Containment Isolation Signal PSIG Pounds Per Square Inch, Gauge RBCCW Reactor Building Closed Cooling Water RBIS Reactor Building Isolation Signal RCS Reactor Coolant System REMVEC Rhode Island, Eastern Massachusetts, Vermont Energy Control RHR Residual Heat Removal RPS Reactor Protection System RWCU Reactor Water Cleanup SDC Shutdown Cooling SPC Suppression Pool Cooling STA Shift Technical Advisor TBCCW Turbine Building Closed Cooling Water TSC Technical Support Center Vac Volts, Alternating Current Vdc Volts, Direct Current l _ _ _ _ _ - _ - _ _ _ _ }}