IR 05000293/1987053
| ML20237C423 | |
| Person / Time | |
|---|---|
| Site: | Pilgrim |
| Issue date: | 12/14/1987 |
| From: | Baranowsky P, Chiramal M, Durr J, Eichenholtz H, Haverkamp D, Knox J, Ruscitto D, Wessman R NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML20237C407 | List: |
| References | |
| 50-293-87-53, NUDOCS 8712210308 | |
| Download: ML20237C423 (101) | |
Text
{{#Wiki_filter:_- _ _-. _ _. _.
U.S. NUCLEAR REGULATORY C0fiMISSION
REGION I
Report No.
50-293/87-53 Docket No.
50-293/87-53 License No. DPR-35 Priority Category C - Licensee: Boston Edison Company 800 Boylston Street Boston, Massachusetts 02199 Facility Nane: Pilorim Nuclear Power Station Inspection At: Plymouth, Massachusetts l Inspection Conducted: November 16-20, 1987 l Inspectors: l J. P. Durr, Team Leader, RI date , D. R. Haverkamp, Assistant Team Leader, RI date $ 44A4ttff) /7/2lf7-P. W. Baranowsky, ean Member, NRR 'date V /2-/ > / 87 M. Chirama,1, Team Member, AEOD .date - , . H. Eichenholtz, Tean Member, RI date Qada nhh, J./L.~ Knox,TeamMember,NRR dat'e CN it /t/qw R. H. Wessman, Team Member, NRR date[
l Approved by: l W. V. Johnston, Director ( Acting) DRS date Inspection Sumary: See Executive Sunmary Areas Inspected: 8712210308 871214 l PDR ADOCK 05000293 Results: G PDR I - _ _ _ _ - _ _ _ -
- _ _ - _ _ _ l
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
l ' Report No.
50-293/87-53 Docket No.
50-293 ! License No. DPR-35 Priority - Category C Licensee: Boston Edison Company 800 Boylston Street Boston, Massachusetts 02199 Facility Name: Pilgrim Nuclear Power Station Inspection At: Plymouth, Massachusetts Inspection Conducted: November 16-20, 1987 Inspectors: ow /,2/N/f 7 (). P. Durr, Team Leader, RI ' dMe $b w am/ /2/t Y O' ? . D. R. Haverkamp, A/sistant Team Leader, RI date k Or'c'...'..'.v ....,, y.
n+ - P. W. Baranowsky, Team Member, NRR ' d' ate ~ Oriut=1Dicu; L;: ,y/4/g 7 M. Chiramal, Team Member, AEOD 'date h^ fx,y
O//4 /0 ' H.Eichenholz,TeamMembg,RI ' date' Orl;ul :g:ae,1:y: p i J. L. Knox, Team Member, NRR ta(e MO /Jkl77 (D' G. Ruscitto, Teamd4 ember, RI ' date ' Orici=1 SIcncc Ir: jg/aff7 R. H. Wessman, Team Member, NRR 'date Inspection Summary: See Executive Summary _ _ - _ _ _ _ -
r- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - l l l , l' i I ' - TABLE OF CONTENTS Page 1.0 EXECUTIVE SUMMARY......................
2.0 INTRODUCTION.........................
2.1 SCOPE OF INSPECTION....................
2.2 TEAM COMPOSITION...........
........... 2.3 SUMMARY OF RECOMMENDATIONS.................
3.0 EVENT DESCRIPTION......................
4.0 EQUIPMENT PERFORMANCE ...................
4.1 ELECTRICAL SYSTEMS.....................
4.1.1 345 kV ELECTRICAL OFFSITE POWER SOURCE.........
4.1.1.1 SYSTEM DESCRIPTION.................
4.1.1.2 OPERATIONS AND FAILURES DURING EVENT........
4.1.1.2.1 STARTUP TRANSFORMER......
...... . 4.1.1.2.2 STUCK BREAKER.................
4.1.1.2.3 SWINGING / OSCILLATING TRANSMISSION LINES....
4.1.1. 2. 4 I N SU LATORS....................
4.1.1.3 REC 0VERY.
.. .............. .. 4.1.1.3.1 WASHING OF SWITCHYARD.............
4.1.1.3.2 BACKFEED THROUGH UNIT AUXILIARY TRANSFORMER..
4.1.1.3.3 RESTORATION OF POWER..............
4.1.1.4 FAILURE HISTORY................
. 4.1.1.5 CHANGES AND IMPROVEMENTS...
......... 4.1.1.6 CONCLUSIONS AND RECOMMENDATIONS...........
1 I _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -
J
_ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ _
.
ll Page 4.1.2 PLANT ELECTRICAL AND PROTECTION SYSTEMS.........
4.1.2.1 SYSTEM DESCRIPTION - 4.16 kV ELECTRICAL SYSTEM....
, 4.1.2.2 SYSTEM DESCRIPTION - REACTOR PROTECTION SYSTEM AND PRIMARY CONTAINMENT ISOLATION SYSTEM,...... .
4.1.2.3 OPERATIONS AND FAILURES DURING EVENT.........
o 4.1.2.4 CONCLUSIONS AND RECOMMENDATIONS............
4.2 EMERGENCY DIESEL GENERATORS..................
4.2.1 SYSTEM DESCRIPTION....................
4.2.2 OPERATIONS AND FAILURES DURING EVENT...........
4.2.2.1 CURRENT TRANSFORMER..................
4.2.2.2 MECHANICAL PROBLEMS..................
4.2.3 CONCLUSIONS AND RECOMMENDATIONS..............
4.3 INSTRUMENT AND SERVICE AIR SYSTEMS...............35 4.3.1 SYSTEM DESCRIPTION........... .........
4.3.2 OPERATIONS AND FAILURES DURING EVENT............
4.3.3 AIR SYSTEM RECOVERY........... ........
4.3.4 CHANGES AND IMPROVEMENTS..................
4.3.5 CONCLUSIONS AND RECOMMENDATIONS...............
5.0 HUMAN PERFORMANCE... ....................
5.1 ORGANIZATION.... ............
........ 5.2 EVENT CLASSIFICATION AND NOTIFICATION.............
, 5.3 USE OF SUPPORT ORGANIZATIONS AND PERSONNEL..........
! 5.4 MANAGEMENT ACTIVITIES AND COMMUNICATIONS... ...
j ... 5.5 OPERATOR ACTIONS.............. ...
.... i , I l -
- ( -
9., l " '.i y 7,,,. g " iy , y -
- b w, *-
(9 Asl ' , y p . ,
i l' t \\* g a
a . i.
- Pace
, .i l . y (( 5.6 PROCEDURE ADEQUACY...... !
.g ............. r 5.7 RADIOLOGICAL ASPECTS...... s.
- .............. .5.8 CONCLUSIONS AND RECOMMENDATIONS.
.............. , ${6.0SAFETYASSESSMENT................ ...
.... j .p-g - [, 6.1 REACTOR SAFETY SIGNIFICANCE OF EVENT.......... ^. . 60 , f.
}s p/ ,,/ .y . . y .. i '6.2 REACTOR SAFETY. IMPLICATIONS AT HIGHER DECAY HEAT LEVELS....
>l'
6.3 STATION BLACK 0UT SAFETY IMPLICATIONS.............
6.4 CONCLUSIONS AND RECOMMENDATIONS................
t - , , APPENDICES ~ APPENDIX A: ENTRANCE INT GVIEW ATTENDEES \\ ' f APPENDIX B: EXIT 0!NTERVIEW TTENDEES ' APPENDIX C: PERSONS INTERVIEWED i e , -fn m ,
s' ' U ,. APPENDIX D: INTEGRATED SEQUENCE OF EVENTS
n . APPENDIX E: BECO RECOVERY SCHEDULES g, APPENDIX F: INSTRUMENT AIR SYSTEM INTERRELATIONSHIPS f ' APPENDIX G: INSTRUMENT AIR SEQUENCE OF EVENTS
APPENDIX H: ACRONYMS AND INITIALISMS t k'q ' hz l . , l y' ' ', ' % i , ., - ,#\\ 'f I 4; .s .c ., 1. , .s b l4 , 'k t-f /
. -,, -_ -. -- -. - - - -_ - -. - - - . - - _ - - - _ _ - - - 'i.
' ' > , ,, t Ll ', <
- 1
' t-l c ' [[ ,
1.0 Erd g tive Summary
The ' Pilgrim Station experienced a tote.! loss of offsite power on November 12,1987, at 2:05 a.m. which extended for a period of 21 hours. The loss of offsite power was initiated as the result of a severe winter storm consisting of gale force winds and wet snow.
The winds whipped trans- ,, mission lines into proximity to each other and snow was packed into high voltage switchyard insuiators causing phase te phase and phase to ground faults on the, offsite transmission lines.
This resylted in the 345 kV
switchyard neeekers open wg and i sola' ting the station.
Both emergency i diesel generators started and ' assumed loads from the emergency buses, i Operator' actions whre correctly taken and the plant was stabilized and }. maintained! n a safe condition throughout the recovery period.
i In conjunction with the switchyard breakers tripping, the station startup traufermer experienced a phase B differential lockout trip. This trans-forrrer is the preferred power source fer outage conditions. and was the only source available for quick recovery of offsite power. Because of the phase B differential lockout trip, the decision was made to perform a series of tests on the transformer to verify no internal fault existed.
This was a major factor in the duration of the loss of offsite power.
At approximately, 11:30 a m.
of that same day, a current transformer y instru:nentatim circuit for one of the diesel generators was determined to have an open circuit and the decision was made to secure the associated diesel to preclude possible ' revere damage.
Once the diesel was secured, additional. equipment fa t iures and malfunctions delayed returning the diesel to operable status until 11:15 p.m. on Nov mber 14, 1987.
The station 'was in an 6 tended outage at the time and reactor safety was never a factw ' due to the viry low decay heat. Hcceve, the plant had equipment cut J service and was configured such %at, under other circumstances R n higher decay heat loads, it couM nave created serious operations.1 inflexibilities.
Fce example, the 23k\\' shutdown transformer, an alternate off site power source, was out of service because of plant modifications and maintenance.
Alsc, two of the three emergency bus supplied instrument air co'npressors were out of service for maintenance work.
Once the diesel was secured, the only running instrument air compressor stopped because it was powered from that diesel.
The other alternate method of supplying power to the station was through backfeeding of the main and the unit auxiliary transfcmers. However, this j method was not readily available due to the need to remove bus links ' , be ween the main generator and the transformers. Ultimately, this was the method used to restore offsite power to the site.
The operational staff resperded well to the event and adequately coped I with the equipe nt failures and malfunctions.
One miscommunication between the on-watch supervisors prematurely secured the diesel generator, but this had no adverse safety effects on the st n ion. The recovery efforts were well planned and reasonably well implemented.once the
i _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _ - _ - _ _._ _ _ _ _ - - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ - - _ _ _ _ _ - _ _ _ - _ _ _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ i.
I
L conditions were properly assessed by station management.
Delays in restoring the diesel to operable status were caused by inadequate maintenance practices before the event.
Because of the frequency with which the Pilgrim Station has experienced losses of offsite power, the complications which resulted during the recovery efforts and the extended period the plant was without offsite power, an NRC Augmented Inspection Team (AIT) review of the event was performed during the period November 16-20,1987.
Although there were no violations of regulatory requirements identified, specific weaknesses were noted that warrant additional licensee and NRC attention.
These are maintenance procedure inadequacies, operator alarm response deficiencies, and equipment malfunctions (see Section 2.3).
Other observations regarding human and equipment performance were made and are referred to the licensee for their consideration.
2.0 Introduction 2.1 Scope of, Inspection In response to the loss of offsite power (LOOP) event at the Pilgrim Nuclear Power Station during the early hours of November 12, 1987, the NRC formed an AIT to determine the sequence, the causes and the safety significance of the event. This was ~ accomplished by establishing a chronology of the event, reviewing equipment performance and plant staff actions relative to the plant design bases, technical specifications and operating procedures.
The NRC team held an initial planning meeting on November 15, 1987, and arrived onsite and performed the inspection during the period of November 16-20, 1987.
Attendees at the entrance and exit interviews are listed in Appendices A and B.
Individuals formally interviewed by the NRC team during the course of the inspection are listed in Appendix C.
2.2 Team Composition The team was composed of a team leader a1d seven headquarters and regional specialists with expertise in plant operations, electrical engineering, instrumentation and controls, risk and safety assessment, and management controls.
2.3 Summary of Recommendations The consensus of the AIT staff regarding the station's response to the loss of offsite power was that the operating staff performed well and that the reactor plant safety was never in question.
The actions taken for recovery were appropriate and well organized.
Coordination and communication between groups was acceptable but improvements are needed to increase , effectiveness.
The coordination of these groups would be substantially enhanced by well defined management guidelines for this type of event i _ _ _ _ _ _ _ _ _
- _-__ - .
(i.e. one in which the formal emergency response organization is not mobilized).
The recommendations presented here are the result of observations made by l the AIT and are not violations of regulatory requirements but do l represent concerns that warrant licensee review. This is a summary of l significant recommendations.
Sections 4.1.1.6 and 4.1.2.4 The operators were not aware of the alarm indicating the reduced voltage on the 345 kV offsite power source prior to the loss of offsite power.
They were also unaware of the alarm indicating the blown fuses in the analog trip system power supply. The failure to utilize these alarms should be reviewed and appropriate corrective actions developed.
l' Section 4.1.1.6 The operation of the startup transformer differential lockout relay was apparently the result of a transient for which the protection was not designed. The transformer did not experience an internal fault and the operation of the lockout delayed the re-energization of the station from-offsite power sources. The actual cause of the differential lockout needs j to be conclusively established.
Section 4.1.2.4 The blown fuses in the analog trip system were the apparent result of a common cause. The cause of this condition should be identified and corrected or determined to be acceptable before the reactor is restarted.
Section 4.2.3 The inoperability of the "B" emergency diese.1 generator during the event resulted from inadequate or incomplete maintenance procedures.
The binding of the prelubrication pump and the leaking fuel injectors could have been prevented from interfering with the recovery operations if adequate procedures for repair and post maintenance testing were employed.
Section 5.8 The plant configuration before the event and the equipment that was out of service for maintenance purpose created operational situations that could have been more serious under other circumstances with substantial decay heat.
Describe what considerations will be made in the future to assure that essential and non-essential equipment removed from service for outage maintenance do not create undue operational inflexibilities.
3.0 Event Description The Boston area experienced a storm beginning about 5:30 p.m. on November 11, 1987, and continuing through the night until about 10:30 a.m. on the 12th of November. The National Weather Service characterized the condi- - _ - _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ -
.. - _ _ - _ - _ _ - ._
tions as gale force winds and wet snow.
For a three hour period during the storm, blizzard conditions existed (See' Table No. 1).
The site
meteorological tower records for the 220' elevation show that wind speeds averaged in' excess of 50 miles per hour during the period of interest, 1:00 to 3:00 a.m. on the morning of November 12, 1987, with gusts peaking to 60 miles per hour (Figure Nos.1 and 2).
When the event occurred, the Pilgrim Nuclear Power Station reactor was shutdown in an extended outage which began in April 1986.
The reactor mode switch was in the " refueling" position and reactor coolant tempera- ' ture was being maintained between 170 to 190 degrees F using the residual heat removal pumps.
Temperature was being maintained at this level in preparation for a reactor coolant system hydrostatic test.
The station offsite power sources are two 345 kV lines, No. 342 and No. 355, which feed power through either the startup transformer or the auxiliary transformer (see Figure No. 3).
In order for the auxiliary transformer to supply power to the station during an outage, the isophase bus links to the main generator must be removed.
This arrangement is referred to' as "backscuttle" by the licensee.
The station can also be supplied by a 23 kV shutdown transformer; however, this source was unavailable for use because preventiva maintenance was in progress and the system was being modified to install a new, third diesel generator.
At 2:05:26 a.m., line No. 342 experienced a phase "A" to phase "C" to ground fault. Because of the ground fault, relay operation opened breaker Nos.103 and 104 at the Pilgrim Station. Breaker No.104 was slow in opening which resulted in the stuck breaker protection operating to open breaker No.105 and sending a transfer trip signal to Auburn Street, opening breaker No. 2130, and Canal Station, opening breaker No. 312.
Breaker No. 412 was previously open before this event.
Line No. 355 was still energized and powering the startup transformer through breaker No. 102.
At 2:05:34 a.m., line No. 355 had a phase "B" to "C" fault.
Breaker No. 1670 at Bridgewater opened and automatically reclosed 0.5 seconds .j later.
In conjunction with breaker No. 1670 opening, breaker No. 102 i supplying the Pilgrim Station startup transformer opened on a transformer phase differential.
During the 0.5 second time required for reclosing breaker No. 1670, the licensee hypothesized that motors, normally energized from the 345 kV transmission lines through the startup transformer and the Pilgrim distribution system, generated a voltage backfeed through the startup transformer to the fault on the No. 355 line.
This backfeed caused the motors to quickly slow down with a resultant frequency decay.
The i frequency decay resulted in an increased voltage / frequency ratio (volts per Hertz overexcitation) to the startup transformer which caused the HU-1 differential protective relaying on the startup transform 2r to activate.
Actuation of differential relaying caused the breaker Nos. 102, 152-104, 152-204,152-304,152-404,152-504, and 152-604 (See Figure No. 4) to trip open.
__________________________-______-____
- _ _ - _ _ - _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ _ _ _ - - _ - _ _ _ _ _ - _ _ _ _ - _ _ _ _. 5~ TABLE NO. 1 BOSTON AREA WEATHER CONDITIONS NOVEMBER 12, 1987 WIND WIND LIQUID TIME DIRECTION SPEED GUST PRECIP.
(Degrees) (Knots) (Knots) (IN) 12:50 a.m.
340
30 .06 1:18 a.m.
330
32 1:54 a.m.
330
33 .11 2:25 a.m.
340
2:54 a.m.
340
30 .13 3.35 a.m.
340
3:53 a.m.
340
35 .12 4:54 a.m.
330
27 .16 5:54 a.m.
330
28 .12 6:54 a.m.
330
26 .10 7:53 a.m.
320
26 .05 Provided by: National Weather Service I ___ _____ ____ ____________________ ____________ -
[
- N
" * re. %)] }%X e ,,g , , , , I J',e
l p , b' - h'$ ;"$. ,, ""a g l .. i , 4.t : i /l3 ih' ". I fi m 'liSiTNt! l '
M'.-sdsl.
l ,7 "'"'
' -
.; ' }l0 lI(I 8h[
$ {l f.
b
O ) 0) " fp*.t:p..;,,,,. i. p : t ...x..
. .. i.
n . . g.
RiRI .' o -" ' . ,1 g .,A ?.l 'I' ~ .. - ] "fQ ' iNNN;N,N_,. didi U ',i gh
.... ,,
- t EN L
gig;.ph .. , . .. - . . ,,,.y p . , s.w.,g; ' ~ -
. .. . .. - ... ., ... C< . hgY h, ,,,$ ' '. , i / O & \\ n' * , g.
m , c-n.r.... 5k.
O l l, h',hF.
, , ' ' in$pg.Rl f '"5 iL i) ? i - - ' hf i.e
' .
'
. . , ,. .
C E b ' }},(l. ; ' P.TO Bl0
' 31 9 " '
. 1. n.p,' W .. - . L.s . T
- .2,
' . ta - < - . , .: p n; .,, r I i . . .'d.,(r ..,. . 'p - n i J < -
.,, , \\ o - .n. ' /,h k dh?
J.
i ll .=
9%%dl ~ ' ,i j ,
- ,
" - ' ' - . . a - , , ph%yl- _ . - - .. . . /. h.n.,.,N "RE \\ d . . . .. s . i u . . . . - - a , d* gun ' I m s
- . ..
WifG: '.L hf;.hy.
uk . - - . > - r et..h.
I l , - - - t ' s ,. , - IhcNkhi.
Ni . h! Ly"j?m jlt QQWik[) ' ' ' , ? $ \\MITD usiG II ' . ,i:: a, ggna-i; ,a n.! ' ._ Mam o j,,g <ca
. i > , . i ' ' {p /h.A,1. k.. p' ,, , ' ~ , l . !: ' W - r ,
V.
fST' 'I', @ [ **fG,h i p .{ s.
.s . , $. i- ~ '-
- .
. bFIGURENO.1 WIND SPEED AND DIRECTION k g - li.f l 2 ', N . ! ' q
' 223 FT ELEVATION [d. W,w'h,v;..- , , O, , ' e
- q, des. *,
I . n - - >
I ,, [ ' i h )' l I ' " r N!.: efi ! I I .. .
.. ' h,. h?N ! l uM ,hb , . o
......... n?Q.s. % -p- . ................. , } _ h( . 1: . ' - l e-,H il g}$:gA.M@ men =l pjf,[ff.' ,,; I 11I!' '9 ' ' ' I ld ~ lllM H M o . l, 1 Ill ' i i e. 1 - . ~ T.,fIl
.* . , . { ' _ !' ' yhM., . @ ,, - - ' ' . , ' !ll
- [(.I$.2
. "U: f;- - ~" - - - - --+ - -?{ ~ ' ' i , ~$ . h.
[ N+ )1 . ._.._.,.,, ,
.f . ' ' s M;,;4II I',,Fj l iB ) . . . . , j(l' ZD 81[d '3) , Ia i .. , l } l l ' ' " ' " '" " " "' ~ " "'~ ' , , ...; .I , _ ?
' s - - -. ' - - - l4 '* if ; . < n
- p
, , .;
- n.
- E ll;7 ' -: _, } ' S.
2I '. ' -[5M ' l " y p ' Ili~']l IM.',l_
.- - i
< . , ,
.. o o .. , r : g u, 3' : o - - . - - . ,
- ' ' n; no
- - , - - , . , .1:.; - ' ' t- ' ' i __ _ . ~ ll;;; . < j a ., -- - 'j - p.
,... ' l : y .-
n ., , g ,. , - <.,..,.... . a...., . , . l_ g.. ' 7) "
I c . , -
n i
. , O L' '
A ' ' --: --
, . , ssfj/r' D i ' ' "=' L",' il; y,iDg;p-, y,:> I s
.14 ., ' , L >1 '- ' A Nf
- .
., ,. 'y
- ,
, ' ' m .- -
- ..
,p- /S4 .f P l r .,,
L * ' '
hIl Til dDl
M I,, j JL 'L
4;i E] B3 ' ' t
, o _ ., Qg :,ty 59WC '
,, - " - r
, -).di -.- ! ! - .e ' - ( ' i
i ~ "'Y ~J"'r j. lf'I'
. ' - . ,, - FIGURE N0.2 h-% 5' ' - {".=- - - '- ""-~~ - -- r-f- (( e.
'I WIND' SPEED AND DIRECTIO:1 33 FT ELEVATION i k l l , , %.. % 'd' e,
- .g.
' '
"" '"
h,.y , m t i;
.. t a na> - ,as;,12ar - . F .. .- - .. ' , _ f; h ' . . wmm , e m en .:: I a n..% , - - - _ - - - - - - - - - - - - - - - - - - - - - - - - _ - - - - - - - - - - - _ - - - - ---
_r c e To _ _ . ,r,e _ _ _ . _ . ,f _ . , f b , '
, T
,o
r c c ] ,7 2_ 2 C e e a
44 ' c n no n2 ee t L3 0N nA a n u o s8 0= s4 e 4 s - 1e e0
s -. To O At 0/ D r s s/ a co
u s
M
1J % % , Jm J E L .Y r A (N S* % E A3 a~e.
- g o R n E M K c Cn t At o A f a4 r8 r 's
h1* e e N s s e e R "i - n R " G E h s.
,
- n i
C O c- . A 'a A ' At o $ M J D' 3 - c 6 Af s
uA am s 's A e s r , O o M s Ag e, r e L . a t
e A nA % M,, - l 0z u q s c e oi r/ q c AI sDp i r o or e , rs r , gg a l.
n 3 aro s1 S r r- - aT D % L T-r 5a C =6 E-c 4 7 t1 e V
R w o n
k M/ s g5 o9 n8
f TA3 c-
ac/ W3 sT Se N iD / O K2 I
3T Gn a A Ks.
.TD n OSR B N MY EI H RRC UGT GLI I I FPS
,l.
,l
w , ' % - . ' - _ a - n - . + , 1[# k - - ,- - - q - p w./v % I) n g V.-
a
f' g E 4 ' , r ' l -' . , f* k
t I )l g 3) . S'9 g
g y *, V t p [', r K - a l h A l - ' V
- 8
$"j A l y
E'3 )
I
A %n.
G g v.
) ' in u ) h<r S , E tt e S fe s U / a.
c B o. D % v L I A , C g-I k, g m R - y a~. 2, g T , sg 4C g .E s L OE ) a N nR le. ' . V Ek eG g R AH gy U6 ,. g aa I ys ) - G ) G1 s. R ,< s
f F4 aF , h
A$ g d
r r e A u tC t l U ~Y f A l c / $ v. 0
y S6 i g - v )i f ' *. - R ! ,
n - , v f ) E - . y s8 . I _ se g* m f l l c,. - _ t s_ i n a v - - s v . N :s J,I.
" kR M a v km U7 t , s s e ) I r b' m 'i s O7 a n m,
at a ' 3 w Uu s G I)1I1 l' ll l
_ _ - _ _ _ - _ - _. _ ._ _
1 The switchyard and.offsite breaker operations resulted in the station ! losing all offsite power and the emergency diesel generators starting to j supply the onsite emergency buses.
All plant systems responded as designed including a full reactor scram and a primary and secondary i containment isolation.
However, efforts to restore the shutdown cooling l after the event by starting the residual heat removal pump resulted in i another primary containment isolation. Subsequent efforts to re-establish shutdown cooling were successful.
Shutdown cooling was not a safety concern at this time because of the very low level of decay heat.
The licensee assessed the event as reportable pursuant to 10 CFR 50.72 and notified the NRC at 2:55 a.m.
As the duration of the loss of offsite power continued into the morning of November 12,'the plant operators experienced additional equipment failures which required operator intervention. At 5:55 a.m. the operators identi-fied blown fuses in the analog trip system panel; three of the four panels l had blown fuses.
At about 6:30 a.m., the plant began to experience instrument air problems because the normal sources of compressed air are powered from nonvital electrical buses.
A single air compressor, powered by the "B" diesel generator, was not able to cope with the demand.
The other two air compressors that are powered by emergency sources were unavailable because of maintenance work that was required to restore them to service.
l At about 10:00 a.m. the plant staff noted that the "C" phase ammeter of the "B" diesel generator was reading zero. This resulted in a recommenda-tion by plant technical personnel with concurrence from BECo Nuclear Engineering Department personnel to secure the "B" diesel to prevent further damage. The concern was that an open circuit existed which could l impress excessively high voltages on the circuit components.
The "B" diesel output breaker was opened at 11:30 a.m., but reclosed automatically due to the presence of a LOOP signal. An operator was dispatched to the diesel generator building and manually secured the diesel by using the emergency keylock switch at 11:35 a.m.
The securing of the "B" diesel generator left the plant with only one source of power, the "A" diesel generator.
Also, it removed the power source for the only remaining air compressor. The loss of all instrument air, although a nuisance, presented no safety hazard to the plant.
The plant management mounted an organized recovery effort employing multiple approaches.
The recovery of offsite power involved attempts to restore the startup transformer, provide power through the auxiliary transformer and an assessment of energizing the 23 kV shutdown trans-former.
In order to support the recovery ef fort, the Nuclear Department Operations Manager administratively staffed the technical support center at about midday.
This allowed the operating shift to focus on plant conditions.
Before the startup transformer could be safely re-energized, it was determined that several tests should be performed to verify no internal fault existed. Complete washing of the high voltage insulators within the switchyard was also thought to be necessary to ensure that fault tripping _ ______________ ___________ __-
- _ _ _ _ _ _.
a problems would not occur. The testing involved Doble Testing of the high voltage side of the transformer, insulation resistance testing of the low voltage side and oil sample analysis.
These tests and cleaning of the insulators were completed at about 2:00 a.m., November 13, 1987.
In parallel to the efforts to restore the startup transformer, actions were initiated to restore power through the a'uxiliary transformer (back-scuttle).
This also required the washing of the high voltage insulators in the 345 kV switchyard in addition to the removal of the isophase bus quick disconnect links between the main generator and the main trans-
former.
This was accomplished and offsite power was first restored by backfeeding through the auxiliary and main transformer at 11:09 p.m., November 12, 1987.
Restoration of power through the 23 kV shutdown transformer was determined not to be a viable method of restoring power to the station due to the extent of work necessary.
This method was not pursued.
An integrated chronological sequence of events is presented in Appendix l D.
4.0 Equipment Performance 4.1 Electrical Systems 4.1.1 345 kV Electrical Offsite Power Source 4.1.1.1 System Description 345 kV Switchyard: The offsite power distribution system consists of a 345 kV ring bus and its associated line connections (see Figure No. 3).
' The ring bus has tie lines to the Canal and Auburn Street (No. 342) and Bridgewater (No.
355) lines.
These incoming transmission lines run adjacent to each other on the same tower for approximately eight miles and then diverge at the Snake Hill Road Tap.
The No. 355 line runs westerly and northwesterly approximately twenty-one miles to the Montaup Electric l' Company's Bridgewater Station.
A tap is made to the No. 342 line at Jordan Road approximately five miles from Pilgrim Station. This line runs approximately twenty-six miles northwesterly to the Auburn Street Station of Montaup Electric Company.
The other branch of the No. 342 line runs southerly for about thirteen miles to the Canal Station of Commonwealth Electric Company.
The tie-ins to the 345 kV ring bus occur via manually operated disconnect switches.
The unit main transformer (X1) ties into one side of the ring bus via manually operated disconnects and the start-up auxiliary trans-former (X4) taps off the opposite side of the ring bus. These connections are isolated by 3,000 amp, 345 kV, air blast circuit breakers (ACB).
The 345 kV transmission lines and ring bus are arranged so that failure of either line will not result in the loss of the main generator, the other 345 kV line, or the startup transformer. The transmission system is protected using carrier relaying on the lines and high speed differential . _ _ _ _ _ _ _ _ - - - _ _ - _ _ - _
- _ - _ _ _ _ _ _ _ _ _ ___
protection on the transformers.
Overcurrent, ground differential and impedance relays protect station components within the ring and protect the grid from faults within the ring.
A fault detected by switchyard y otective relaying will trip breakers adjacent to the fault as long as the bus is energized.
The 345 kV breakers are controlled directly from the main control room.
Breaker position, 345 kV line voltages and other parameters are monitored in the main control room.
Unit Alternating Current (AC) Power Source: The station main generator provides power through the isolated phase bus at 24 kV to both the main transformer (XI) and the unit auxiliary transformer (X3). The generator voltage is stepped up through the main transformer to the 345 kV ring bus voltage.
The generator voltage is stepped down to 4.16 kV through the unit auxiliary transformer for the station auxiliary power distribution system (see Figure No. 4). The unit auxiliary transformer has two 4.16 kV windings, X and Y.
The X winding feeds 4.16 kV switchgear buses A3 through A6 while the Y winding feeds 4.16 kV buses Al and A2.
Refer to paragraph 4.1.2.1. for details on the 4.16 kV system.
Preferred Alternating Current Power Source: The preferred ac power source (startup transformer) provides a source of offsite ac power to the auxiliary power distribution system which is adequate for the startup, operation and shutdown of the station.
The startup transformer (X4) supplies power to the station auxiliary power distribution system whenever the main generator is offline.
After the main generator has been synchronized to the 345 kV system and has been partially loaded, the auxiliary power distribution system is manually transferred from the startup transformer to the unit ac power source.
Automatic fast transfer capability is provided to restore the preferred ac power source to the station auxiliary power distribution system in the event that the unit ac power source is lost for any reason.
Should power be interrupted to the preferred ac power source due to a double 345 kV line fault, it will be automatically rest %j when the line breakers reclose after the fault is cleared and the lines are re-energized.
This automatic reclosure is designed to prevent both 345 kV breakers from reclosing at the same time.
The breaker control for the preferred ac power. source is interlocked to prevent interconnection with the secondary ac power source. The preferred ac power source may be synchronized and interconnected with the unit ac power to permit live source transfer following synchronization of the main generator.
4.1.1.2 Operations and Failures During Event At the time of the initial loss of off site power on November 12, 1987, the 345 kV ring bus was fully energized with all breakers closed.
The startup transformer was supplying all 4.16 kV buses and both diesels were in standby. The main transformer was isolated at the ring bus disconnect switch, T-930, and the main generator iso phase disconnect links were closed. All 4.16 kV buses were isolated from the unit source. The 23 kV secondary source from the shutdown transformer was not available due to _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.
_ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -
l
maintenance which included the tie-in of the new, third " blackout" diesel generator.
At 1:27 a.m. on November 12, a 345 kV line low voltage was alarmed on the l main control room computer logger. Analog voltage indication was 338.73 ' kV. The value remained below the alarm level of 341.50 kV until all power was lost at 2.06 a.m.
The nuclear operations supervisor (N05) and nuclear . watch engineer (NWE) indicated that they were not aware of the alarm.
It i is not' clear whether any other operator was aware of the low voltage I condition; however, no actions were taken to initiate procedure. 2.4.144 " Degraded Grid Voltage" which would have included monitoring 345 kV bus voltages and contacting Rhode Island, Eastern Massachusetts, Vermont Energy Control (REMVEC) about grid stability.
More specifically, a check ! of the 345 kV bus voltage recorder in the control room clearly shows the i above mentioned voltage drop which was sustained until the complete loss of offsite power. Although in this case missing the alarm had no impact on the overall sequence of events, the failure of the operators to consult the appropriate procedures in response to the computer alarm.is of concern to the NRC.
At 2:05:26 a.m. on November 12, 1987, one of two 345 kV transmission lines (No. 342) connecting the Pilgrim Station switchyard to the Boston Edison transmission network, experienced an "A" phase to "C" phase to ground fault (see Figure No. 3).
As a result of this fault on the No. 342 line, protective relaying operated causing breakers No.103 and No.104 to trip open to isolate the fault. Also, because No. 104 breaker was indicated slow to open, backup or stuck breaker protective relaying operated causing a transfer trip signal to be sent to breaker Nos. 103, 105, 2130, 412, and 312. The transfer trip signal caused breaker Nos. 105, 2130, and 312 to trip open and caused the automatic reclose feature on breaker Nos. 103, 105, 2130, and 312 to be removed or locked out.
For a short time following isolation of the initial fault on the 342 line described above, the Pilgrim Station was energized from the remaining 345 kV transmission line (No. 355) through breaker 102 and the startup transformer. At 2:05:34 a.m. (7 seconds following the initial fault on the 342 line) the 355 line experienced a "B" phase to "C" phase fault. As a result of this fault on the No. 355 line, protective relaying operated causing breaker 1670 to trip open and to reclose 0.5 seconds later, i During the 0.5 second time required for reclosing breaker No. 1670, the licensee hypothesized that motors, normally energized from the 345 kV transmission lines through the startup transformer and the Pilgrim distribution system, generated a voltage backfeed through the startup transformer to the fault on the No. 355 line.
This backfeed caused the motors to quickly slow down with a resultant frequency decay.
The frequency decay resulted in an increased voltage / frequency ratio (volts per Hertz overexcitation) to the startup transformer which caused the HU-1 differential protective relaying on the startup transformer to activate.
, Actuation of differential relaying caused the trip opening of breaker Nos.
102, 152-104, 152-204, 152-304, 152-404, 152-504, and 152-604 and the loss of offsite power to the Pilgrim station from the Boston Edison transmis-sion network.
- - _ _ _ -. - - - - - _ _ _ _ _ _ _ _ ___j
_ _ _ _ - - _ _ _ -- -__
4.1.1.2.1 Startup Tra. 'f;,ner The station lost all ot 7 te power at 2:06 a.m. when breaker No.102 opened. This was due to i,ie actuation of the B phase differential relay of the Startup Transformer.
< The Startup Transformer is the immediate access for offsite power to the station onsite electric distribution system, i.e., the preferred power source. The transformer is a three winding, three phase, 345-4.16-4.16 KV unit, with the primary side (H winding) rated 20-26.6-33.3 MVA and each secondary (X and Y) rated 10-13.3-16.65 MVA, OA/FA/FOA..It is located in the switchyard at the Southern Boundary.
It is connected, on the primary side', to the 345 KV switchyard ring bus between ACB 102 and ACB 103. The transformer secondary sides are connected to the station 4.16 kV buses through underground cables, the X winding supplying buses A3, A4, A5, and A6, and the Y winding supplying buses Al and A2. (see Figure No. 3.) The transformer is designed to supply the power requirements of the station's two emergency service buses A5 and A6 (feeding safety-related loads), and the four normal service buses A1, A2, A3, and A4.
The Startup Transformer is used to supply the 4.16 kV buses during normal plant startup and shutdown. After the main generator has been synchron-ized to the 345 kV system and loaded, the 4.16 kV buses are individually transferred from the Startup Transformer to the Unit Auxiliary Trans-former.
In the event the Unit Auxiliary Transformer is lost, the 4.16 kV buses are automatically transferred to the Startup Transformer, the pre-ferred power source.
Prior to the operation of the startup transformer differential relay, the transformer was supplying all the required electrical loads of the station 4.16 kV and 480 V systems. When the B phase differential relay (187-48) actuated it energized the startup transformer lockout relays (186-4 and 186-4X), which opened switchyard ACB No. 102 and all the 4.16 kV startup transformer feeder breakers to buses A1, A2, A3, A4, AS, and A6.
The relays also actuated an alarm on panel C-3 in the control room. On loss of power to buses A5 and A6, Diesel Generator A and Diesel Generator B automatically started and energized the respective buses as designed.
Differential protection of the startup transformer is provided by three single phase Westinghouse HU-1 transformer differential relays (designated 187-4A, B, andC). These relays provide protection for faults internal to the differential zone extending from the primary side bushings of the transformer to the 4.16 kV startup transformer feeder breakers at buses Al through A6. The differential relays are not designed to operate for faults external to the zone; neither do the relays operate on magnetizing in-rush currents associated with energization of the transformer.
For moderate internal faults, the differential unit and harmonic restraint unit of the differential relay will operate tripping the relay. Operation of these two units, in turn, energizes the indicating contactor switch (ICS) unit which completes the trip circuit through the transformer lock- - _ _ _ _ _ _ _ - _ _
_ ___ _ _ _ _ _ _ _ - _ _ - _ _ _ _ _ _ - -___ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ \\
out relay. An indicator target on the HU-1 relay (the right target, when viewed from the front) will also drop.
For heavy internal faults, the indicating instantaneous trip (IIT) unit will operate and complete the trip circuit to the lockout relay.
In this instance the left indicator target will drop.
During the initial stages of the Pilgrim LOOP, startup transformer differential relay for phase B actuated, alarmed in the control room, tripped ACB No. 102 and caused the loss of power to the station.
The control room operators recall a distinct time lapse between the opening of ACB No's. 103, 104, 105 and the opening of ACB 102 (due to the operation of the differential re'l ay). The right indicator target on the differential relay was noted by the operator, indicating ICS operation.
After washing and cleaning the transformer, the primary and secondary sides, including the 4.16 kV cables and breakers were tested. The transformer oil was also tested. The transformer passed all tests and, on November 14, 1987, the transformer successfully supplied power to the station. Based on success-ful tests and operation of the transformer, it can be concluded that either the relay misoperated or spuriously operated for a fault outside the zone of protection.
An analysis by the licensee on the relay operations data at Pilgrim and other substations on the 345 kV system during the snowstorm of November 12, 1987, presented the following reason for the transformer differential relay operation: The motors at Pilgrim Station, normally energized from the startup transformer generated a voltage backfeed to line No. 355.
The motors quickly slowed down with a resultant frequency decay.
This resulted in an increased voltz per-hertz to the startup transformer.
As a result, the HU-1 transformer differential relay operated to open breaker No. 102.
The analysis done was based on an assumption that the transformer differential relay actuated after the 2:05:34 a.m., line No. 355, phase B to C fault.
Based on the above data, the transformer differential and the subsequent opening of ACB 102 could have occurred before the fault on line No. 355.
The differential could have operated on a fault on the high voltage bushing of the transformer.
A previous event in which the startup transformer phase C differential relay actuated for a fault on the high voltage side is described in Pilgrim LER 83-045.
4.1.1.2.2 Stuck Breaker The control and protection logic for each breaker in the ring bus at the switchyard is designed such that if a breaker receives a trip signal and does not open or is slow in opening, the stuck breaker protection scheme comes into action.
This feature will trip the two adjacent breakers in _ _ - __
- - _ _ _ _ -_ __ _ . _ _ _ _ _ _ _ _. _ _ _ __
the switchyard thus protecting the ring bus and the grid from faults not initially isolated. The scheme, in addition, sends a transfer trip signal to adjacent breakers at the other end of the transmission lines at Canal, Auburn or Bridgewater stations to further isolate the initial fault.
At 2:06 a.m. on November 12,1987, the 345 kV line No. 342 had a phase A to phase C to ground f ault, which through protective relaying opened ACB No. 103 and ACB No.104 at the switchyard (See Figure No. 3).
However, because of the indicated slow response of ACB No.104,.the stuck breaker protection operated, as designed, to open ACB No. 105 at the Pilgrim switchyard and send a transfer trip signal to open breaker No. 2130 at Auburn Street station and breaker No. 312 at Canal Street.
i ! The licensee is investigating the root cause of the slow response of ACB 104 to determine if the problem is with the main breaker mechanism, the auxiliary contacts, or the associated relays, and whether the problem is related to the cold weather.
Corrective actions based on the findings of the investigation should improve the reliability of switchyard breaker operation.
4.1.1.2.3 Swinging / Oscillating Transmission Lines At 2:06 a.m. on November 12,1987, the 342 line experienced an "A" phase to "C" phase to ground fault. Following this initial fault on the No. 342 line, the No. 355 line experienced a "B" to "C" phase fault. These faults on the Nos. 342 and 355 lines were believed to be caused by snow and wind.
With snow buildup and dropping of snow at different times, the various phase lines and the static line would be swinging with the wind and oscillating at different rates causing them to come into proximity with each other.
When the lines came near enough, a fault was detected by protective relaying which isolated the lines.
4.1.1.2.4 Insulators At 2:06 a.m.
on November 12, 1987, the Nos. 342 and 355 lines were isolated by protective relaying due to faults on the lines.
Following isolation of the Nos. 342 and 355 lines, the insulators located in the Pilgrim Station switchyard were observed to be packed with snow from top to bottom on the sides facing the northeast wind.
Also, the insulator skirts were imbeded in snow. At this time with their snow covering, the insulators would have been unable to perform their design function of insulating the 345 kV lines from ground.
The snow would have allowed, if energized, the 345 kV lines to fault to ground.
The insulators required to re-energized the startup transformer were washed, cleared of snow, and returned to operable status at 1:30 p.m., approximately 11.5 hours following the initial isolation of the fault on the Nos. 342 and 355 lines.
The insulators required to re-energize the main and unit auxiliary transformers for backfeed were washed, cleared of snow, and returned to operable status at 8:00 p.m., approximately 18 hours following the initial isolation of the fault on the Nos. 342 and 355 lines.
--__-____ _
g
4.1.1.3 Recovery Subsequent to th'e loss of offsite power at 2:06 a.m., operations and electrical maintenance department's efforts involved investigating the cause of all switchyard breakers tripping open and the lockout of the-startup transformer.
It was recognized from the very beginning that washing of the switchyard would be required due to the severity of the storm and snow accumulations.
The recovery activities involved with washing of the switchyard are described below. At the morning production meeting, which was chaired by the Outage Manager at 7:30 a.m., the Modifi-cation Management Group was directed to determine the feasibility of returning the 23 kV shutdown transformer to service.
By approximately 10:00 a.m. a recovery team was formed under the direction of the Outage Manager (Maintenance Section Manager).
This team consisted of members of the following licensee groups: outage management, station management, corporate oversight, nuclear engineering department, administration, technical, and maintenance. Individuals were tasked with specific assign-ments, which included plant walkdowns to develop estimates of effort necessary and sequences needed to complete the various recovery options.
Where applicable, plant procedures were reviewed during the walkdowns. At this point, the licensee concluded that the return of the shutdown trans-former to service was not feasible, and no further consideration of this option was warranted.
The options being considered consisted of: 1) returning the startup transformer to service, and 2) backfeeding (back-scuttle) the switchyard through the main and unit auxiliary transformers.
However, at approximately 11:30 a.m., the "B" diesel generator was removed from service to investigate the loss of a "C" phase current transformer instrumentation circuit.
Planning efforts to return the diesel generator to service were initiated and became one of the options to augment the "A" diesel generator, which was the single source of power available to
operate plant equipment.
The results of the licensee's initial planning and coordination efforts for recovery of the plant from the loss of offsite power is depicted on the schedule of events released at 1:00 p.m. on November 12, 1987, which is enclosed in Appendix E of this report. A subsequent revision to the schedule was issued by outage management later in the day at 7:00 p.m., and on November 16,1987 a schedule of the actual implementation of the recovery plans was released.
These documents are also included in Appendix E.
As part of the recovery effort, the team leaders assigned responsibility for the various parallel options being implemented for power recovery were tasked with reporting their status on an hourly basis to the Technical Support Center. In addition, the Manpower Coordinator was tasked with the , responsibility of determining what personnel were onsite, their hours J worked, and the disciplines represented.
A sleeping area and meals were provided to workers involved in the recovery ef fort.
The AIT concluded that the planning efforts conducted by the Outage Management Group were a positive factor in coordinating the recovery effort.
However, there appeared to be a lack of participation in the recovery planning process by the operations group.
l _ - __ ___ _ _ ____ _
_ _. _ _ _ _ __ _- ,
l 4.1.1.3.1 Washing of Switchyard i Directly following the loss of offsite power, the electrical supervisor on shift assessed the switchyard condition.
Because of high winds and i driving snow conditions, activities were limited to those involved in I preparing for the hot washing of the switchyard. This action consisted of . moving the hot washing machine from the maintenance shop to the switch-l yard, and connecting the unit to water and power.
As requested by the l Nuclear Watch Engineer (NWE) the Senior Electrical Engineer in the Main-tenance Group arrived onsite at 4:00 a.m.
By 4:30 a.m. his walkdown of the switchyard determined that 1) the startup transformer had 3"-4" of heavily l packed snow that was bridging the gaps of the insulators; 2) the switch-l yard insulators for disconnects and ACB's were heavily coated with snow; 3) main and unit auxiliary transformers had snow buildup but did not , require washing at that time; 4) the switchyard would need selected l washing to insure reliable reconnection of offsite power; and 5) the pre-ferred method to restore offsite power to the station would involve j returning the startup transformer to service.
The switchyard was washed in accordance with maintenance procedure 3.
M.3-20, Rev.
2, Live High Pressure Wet Washing Procedure.
Switchyard configuration at the initiation of washing consisted of lines No. 355 and No. 342 de-energized, all disconnects closed except T900 and T-930, and ACB's No.s 102 through 105 open. Although all preparation for washing was complete at 6:00 a.m., the required authorizations from the NWE and REMVEC were not provided until 6:53 a.m.
The sequence of washing started on the line side of the T901 disconnect to the startup transformer bushing and 103A disconnect (see Figure No. 3). At approximately 1:30 p.m. this phase of the switchyard washing was complete, which allowed power to be restored to the 355 and 342 lines.
' Following the initial washing ef fort, the maintenance personnel moved the washing equipment to the other side of the switchyard to support the effort to restore power by backfeeding through the unit auxiliary trans-l former.
This phase of the washing effort was initiated at approximately ' 5:00 p.m.
and completed at 8:00 p.m.
The equipment washed consisted of disconnects 105A, 105B, 104B, T931, and ACB 105.
Throughout all washing i efforts, the equipment being washed was de-energized. A warming trend in temperature and reduction in wind strength resulted in the second phase washing efforts being accomplished in less time than the first phase (i.e., three hours vs. six and one half).
j 4.1.1.3.2 Backfeed Through Unit Auxiliary Transformer < Because of the differential phase "B" fault lock out that occurred in the j startup transformer, the licensee consioered it prudent to perform ) electrical tests on the high and low voltage sides of the transformer.
In addition, an oil and gas sample was obtained from the transformer for i analysis. As shown on the 1:00 p.m. schedule of events in Appendix E, the restoration of power utilizing the startup transformer was initially I estimated to be accomplishable by 5:00 p.m.
However, the recovery team - _ _ _ - _ _ - _ _ - _ _ _ _ - _ _
] r )
i determined that it would be advisible to proceed with efforts to use the backfeed option for recovery of power to the station buses in the unlikely event of damage to the startup transformer, i The backfeeding of the station is an abnormal operation that is performed when the unit is shutdown, and the startup transformer is to be taken out
of service.
The unit auxiliary transformer.is used to supply 4.16 kV loads.
To accomplish this, the main generator iso phase bus quick dis- , connect links are removed, and the main and unit auxiliary transformers I are energized from the ring bus in the switchyard through ACB's No.104 and 105; thus, supplying power to the 4.16 kV system. However, the stuck breaker trip of ACB 104 earlier,in the day would allow only the use of ACB No. 105.
Efforts to backfeed the 4.16 kV buses through the main and unit auxiliary transformers were controlled through Maintenance Request (MR) 87-732.
This MR implemented maintenance procedure 3.M.3-9,. Rev. 3, Iso-Phase Bus { Quick Disconnect Links Removal / Main and Unit Auxiliary Transformer Bcck-scuttling.
Both FSAR Section 8.2.1.2 and Technical Specifications Section l 3.9, Bases, make reference to the ability to backfeed power from the switchyard to the station auxiliary loads by isolating the generator from the main and unit auxiliary transformers.
The original backfeed plans envisioned power availability for the 4.16 kV bus to be completed by approximately 2:30 a.m.
on November 13, 1987.
However, the following factors resulted in availability of this power source by 11:00 p.m.
on November 12, 1987: 1) Staging was already in place under the iso phase bus duct; 2) The person directing the quick disconnect link removal (an NED engineer) was experienced in this task; and the related isolation of equipment required by the procedure; and 3) switchyard washing was completed in a timely manner. The tagging required ' by the procedure to allow disconnect links removal was accomplished at approximately 3:00 p.m.
and the links open at 5:00 p.m.
Even though washing of the switchyard would continue until 8:00 p.m., the critical path would be completion of the procedural requirements in 3.M.3-9 that would align plant equipment to support the backfeed. The main transformer disconnect T930 was ready for closure at 10: 50 p.m.
but, because of a linkage problem the disconnect could not be closed until 10:58 p.m.
Additionally, the initial attempt to close ACB No.105 failed because a synchronization switch was not placed in the "0FF" position.
At 11:09 p.m., ACB 105 was closed and backfeed power became avaliable to energize the 4.16 kV station auxiliary buses.
4.1.1.3.3 Restoration of Power Once the backfeed of power to the main / unit auxiliary transformers was complete, the plant operators initiated re-energization of the 4.16 kV auxiliary buses and their associated 480V buses.
The sequence of the auxiliary bus restoration was A6, A4, A3, A2, and A1.
Auxiliary bus A5 continued to be supplied by the "A" diesel generator.
- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.._ _
- _ _ _ _ _ _ - _ _ - - _ _ -. - - _ _ - - --- _ _ _ _ _.
I
The foregoing discussions reflects BECo's restoration of power via the backfeed option. A parallel effort, as shown on the schedule of events in Appendix E, involved the return of the startup transformer to service.
The NED recommended that testing should be performed on the startup trans-former prior to its re-energization.
Plant personnel had noted that the startup transformer miscellaneous alarm had not been activated.
This alarm provides monitoring of transformer parameters such as: oil level, . oil flow, hot oil temperature, combustible limits, and sudden pressure.
I The faut that this alarm was not actuated, combined with the knowledge from past events at the station that close-in faults had actuated the startup transformer differential protective relaying which resulted in lockout actuations, provided an indication that the transformer had not experienced damage.
Because plant conditions were stable with extremely low fuel decay heat, the licensee directed that the transformer's secondary side windings be insulation resistance tested and the primary side have a Hi-Pot test conducted. Although this testing was expected to be completed by 5:00 p.m.
on November 12, 1987, unexpected problems developed that ultimately delayed the availability of this power source.
The first problem was a determination that Hi-Pot testing could not be performed due to unavailability of equipment at the station. This situa-tion was rectified by the station's request for Doble testing equipment and personnel be brought to the plant.
The Doble test equipment was onsite and setup by approximately 6:30 p.m. Once the crew was onsite, it was determined that the lightning arrestors needed to be disconnected.
These two conditions added significant delay in testing of the trans-fo rme r.
There was no existing station procedure to control the Doble testing on the transformer.
All testing performed was controlled and documented under MR 87-731, TP 86-113, Rev.
O, Meggering Procedure and 3.M.111, Rev. 1, Routine Maintenance.
Once the startup transformer was satisfactorily tested, which was approximately 1:30 a.m.
on November 13, 1987, plant personnel continued activities to energize the transformer and make it available as a power source for the station auxiliaries. By 3:35 a.m., the startup transformer was energized and available for service. Because the plant was in an off-normal lineup at the time, the licensee developed and performed an Opera-tions Review Committee review on November 14, 1987 for Temporary Procedure (TP) 87-252, Rev.
O, Station Power Restoration from November 12, 1987 Incident.
This TP provided guidance to the Operations Department to restore the plant to a configuration such that all station power was being supplied by the startup transformer.
The initial conditions were that buses A1, A2, A3, A4, and A6 were being supplied from the unit auxiliary transformer via backfeeding of the main transformer.
Bus A5 was being powered from the "A" diesel generator. To accomplish these tasks, the procedure reiterates the normal recovery from the unit auxiliary transformer as outlined in Procedure 3.M.3-9 for Bus Al-A4 and A6.
For recovery of Bus A5 from the "A" diesel generator, steps which were outlined in Station Procedure Number 2.4.16 were used.
, - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
-_ _ _.._.___.__ _ . _ _ _ _ _ _ la
At 4:35 a.m. on November 14, 1987, the station power was normalized on the startup transformer and the backfeed was secured by opening ACB 104 and 105 in the switchyard.
I An estimate was made by the AIT as to what a reasonable recovery time would have been had plant conditions warranted an expedited restoration of offsite power.
It appears that: 1) with the startup transformer differential lockout being perceived by the licensee as resulting from a condition that was not damaging to the transformer itself; and 2) offsite power reliability necessitated switchyard washing, it was concluded that between 1:00-2:00 p.m. on November 12, 1987 the startup transformer could have been returned to service.
4.1.1.4 Failure History Since the Pilgrim Station went into operation in June 1972, the station's 345 kV offsite system has experienced twenty events, including the November 12, 1987 event, which have caused loss of 345 kV offsite power to the station's onsite electric distribution system.
Of these twenty
events, five were caused by a single line outage occuring during the same time frame that equipment was out of service for maintenance or modifica-tion, five were caused by lighting, four by snow and wind, three by insulator salt contamination during ocean storms, and three by other miscellaneous causes.
In addition to the 345 kV offsite system, the Pilgrim Station maintains a secondary 23 kV offsite power line from a neighboring utility to the onsite safety related distribution system.
This 23 kV offsite line is designed to automatically carry the stations safe shutdown loads given loss of the offsite 345 kV and onsite diesel generator systems.
The unavailability of this 23 kV line and the 345 kV system will cause a total LOOP event at the station.
The 23 kV line was unavailable at the Pilgrim Station during four of the twenty events described above.
Three (3) unavailabilities were caused by snow and wind and the remaining unavailability (caused by its being tagged out of service for maintenance and modification during the November 12, 1987 snow and wind storm) may also have been caused by snow and wind. A total of four LOOP events have, thus, occurred over the past sixteen years at the Pilgrim Station.
The frequency of total LOOP's to date at the Pilgrim Station, is 4/16 or 0.25 per reactor year (counting the November 12, 1987 event).
, 4.1.1.5 Changes and Improvements Since the Pilgrim Station went into operation in 1972, the licensee has initiated and completed a number of changes for the purpose of improving the overall reliability of the 345 kV offsite syste.. _ _ _ - _ _ _ _ _ _ _ _
On March 15, 1979 the licensee initiated a design modification to the 345 kV transmission system in order to reduce the number of double line outages (with consequent reactor trip) being caused by lightning.
This modification was completed on March 30, 1980. Prior to modification there had been three double line outages due to lightning.
Follow modification, there have been no double line outages due to lightning.
In 1980, the insulators on the Nos. 342 and 355 lines subject to salt contamination, were replaced with RG units.
This type of insulator has a resistance glaze which produces a leakage current of about 0.5 milli-Amperes / unit when connected to an energized line. The design principle is that the leakage current will keep the insulator warm resulting in more uniform and rapid drying during fog or mist condition.
Hence, the high voltage gradient across the insulator's surface, normally associated with the uneven drying on contaminated conventional insulators, is eliminated and in turn the possibility of flashover is reduced.
The licensee believes i that this modification has significantly reduced transmission.line flashovers ' caused by surface contamination since 1980.
A program for routine inspection and periodic washing of the switchyard insulators was initiated to reduce flashover incidents caused by salt contamination buildup.
The licensee found that this technique has been effective in reducing flashovers attributed to slow buildup of contamina-tion, but has not been as effective for a severe buildup over a short time.
Thus, the licensee investigated the use of a new silcone elasto-meric coating for application on switchyard insulators.
Industry experience with this new coating has shown that it will give years of flashover protection on transmission lines even in locations with heavy industrial contamination, dust, pollution from highways, and salt fog.
The licensee completed installation of this coating on all but two switch-yard insulators in April 1987.
Operating experience, limited to the November 12, 1987 snow storm, gave a preliminary indication, based on an observed reduction of insulator noise, prior to the 2:06 a.m. loss of power, that the new coating will be effective in reducing flashover due to salt contamination.
Other changes which have been or will be completed before plant restart include modification of the startup transformer fast transfer logic, modification to protective relaying, and installation of monitoring equipment for better analysis of 345 kV system faults and breaker opera-tion.
4.1.1.6 Conclusions and Recommendations Conclusions Changes implemented by the licensee for improvement of reliability of the 345 kV offsite system reduce, to the extent practical, the likelihood of 345 kV system failure due to weather conditions.
_-___
_ _ _ _ _ _ _ _ _ _
The stuck breaker protection functioned as designed.
The licensee is investigating the root cause of the indicated slow response of ACB 104 to , determine if the problem is with the main breaker mechanism, the auxiliary ) contacts, or the associated relays, and whether the problem is related to ! the cold weather.
Corrective actions based on the findings of the investigation should improve the reliability of switchyard breaker operation.
Recommendations ! Misoperation or spurious operation of the differential relay for faults outside the zone of protection lead to lockout of the startup transformer requiring thorough investigation of the unit before returning it to service and loss of access to offsite power during that period.
Hence, the root cause of the relay operation must be determined and corrected in order to increase the reliability of fast access to the offsite power source. The licensee is continuing the investigation of the relay opera-tion on the initial assumption of increased volts per-hertz being the root cause.
However, the licensee has not investigated other possible reasons for the operation of the relay.
Current transformer performance, cali-bration checks and other operation and maintenance actions recommended by the relay manufacturer should be considered.
i Section 4.1.1.2 discusses the failure of plant operators to respond to the degraded 345 kV grid voltage alarm.
Similarly, an observation in Section 4.1.2.4 of this report identifies an annunciated alarm that was not factored into the operator's analysis of the spurious primary containment isolation signal upon restart of the RHR pumps.
These two events should i be reviewed and a determination made regarding the appropriateness of the operator's responses to these alarms and if a training deficiency exists.
) Procedure 3.M.3-9 should be revised to reflect operational consideration for backfeeding with off normal electrical system lineups.
Develop procedures that describe and control testing activities on switchyard equipment and transformers.
4.1.2 Plant Electrical and Protection Systems 4.1.2.1 System Description - 4.16 kV Electrical System The auxiliary power distribution system (APDS) supplies the ac power required to safely shutdown the reactor and maintain it in the shutdown condition.
The APDS also provides power to operate all auxiliaries necessary for safe startup, operation and shutdown of the station.
The six 4.16 kV buses (Al through A6, see Figure 4) comprise both the emergency service and normal service portions of the APDS.
The two emergency service buses, A5 and A6, supply power to essential loads _ _ _ _ __ _ . _ _ _ _ _ _ _ _ _ - _ -
- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ - _ _ _ _ _ - -
1 i required during operational transients and accidents.
The four normal service buses, A1, A2, A3, and A4, supply power to other station auxiliaries requiring ac power during plant operation.
The power _ supply to. the six 4.16 kV Vac auxiliary buses during normal operation is either from the unit source (unit auxiliary transformer X3) , or the preferred source (startup transformer).
Additionally, emergency l buses A5 and A6 may also be supplied from the standby power source (diesel generators) or the secondary source (shutdown transformer, X13).
The eight 480 Vac buses, B1 through B8, are also divided into normal service buses and emergency service buses.
The essential 480 Vac auxiliaries required during operational transients and accidents are all supplied from the three emergency service buses (B1, B2, and B6).
The l five normal services buses (B3, B4, B5, B7, and B8) supply power to other 480 Vac auxiliaries required during plant operations.
Secondary AC Power Source: The seconda ry ac power source provides an additional source of power to the emergency service portion of the auxilia ry power distribution system to permit portions of the 345 kV system to be removed from service for inspection, testing, and mainte-nance.
It is connected to a 23 kV transmission line (No. 72) which is supplied from Commonwealth Edison Company's Manomet Substation.
This substation is supplied by an approximately twelve mile long,115 kV line from the Horse Pond Switching Station.
The incoming voltage is reduced from 23 kV to 4.16 kV by the shutdown transformer (X13).
The shutdown transformer may be connected to the emergency service buses A5 or A6.
Standby AC Power Source: The standby ac power system consists of two 2600 kW, ALCO emergency diesel generators (EDG's), each of which is capable of providing sufficient power to satisfy the loads on its respective bus (A5 or A6). Control power required for the startup and operation of each EDG is supplied from the 125 Vdc system.
Other auxiliaries necessary to ensure continuous operation are supplied as required from the EDG through the emergency service buses.
Each EDG with its associated auxiliaries, connection to 4.16 kV emergency switchgear, control system, and distribution of power to the various safeguard loads, is segregated and sepa rated from the corresponding systems of the other EDG.
Each EDG is operated independently of the other.
See Section 4.2.1.
4.1.2.2 System Description - Reactor Protection System and Primary Containment Isolation System , { Two motor generator (MG) sets normally supply ac power to the RPS (see ) Figure 5).
RPS MG set A is supplied ac power from 480 V, MCC B-23 which is connected to 480 V bus B3. The RPS MG sets A and B supply power to 120 V RPS buses A and B, respectively. Alternate power to either RPS buses is from 480 V MCC B-10 (through transformer X-20) off 480 V bus B6. Controls and interlocks are provided such that only one RPS bus can be supplied by _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ -
. 480V MCC W-23 480V MCC B-22 _ 480V MCC s-10 (4 ......S . .,-,,,.S , (t v " S.-,... . RPS h h [" h RPS MG SET A M). . (U) - MG SET V , I I I ,8;Y c.y' 1 ,0.TSKVA sA> <Ai dSO/,89V 18.T S K V J ~S m,, S._ (.y, _ ..;Y coH ' "* VR VR I I VOLTAGE VOLTAGE O s.d e.d O ~ ~ FREQUENCY FREQUENCY q, , PROTE CTION PROTE CTION
r:
I I j PwR SUPPLY PwR SUPPLY PwR SUPPLY f PROTECTON PROTECTION PROTECTON
ASSEWSLY ASSEMSLY ASSEMBLY , ) ,h~~. EPA 1 __ J EPAf ._ j EPA 3 ] SEISMIC UPGRADE ASSEMsLIES l I I , ' Pwm SUPPLY PwR SUPPLY PWR SUPPLY i PROTECTCN PROTECTON PROTECTCH k ASSEMBLY ASSEMBLY ASSEMBLY ...) EPA 2 .. JEPA6 . J EPA 4 ' ... - . . - - -... -... - -. . .. ... - . . .. < MSnuttiy OperSted . .
Cs1A Cs2A Cgra paie ~ ~ ~ ~ ~ ~ ~ MbH ANSCAL #dERLOCK "" ~~ ~ 3~ ~ ~
L...--....- -... TR ANSFER 8 WITCH C-511 l , ... - - -... - - -... -.. - - -..-...J fs @ @ O.
0,, O.
. ,, ,, RPS SUS 'A' _ RPS SUS 'S' OII E e-CB1A CB2A C828 CB18 . C-511 TRIP LEVERS :@ Determine. which Suppil. RPS 'A' hDet.tmine. which Bu. Rec.lv.. Stensby Power @ D.i., min...n..h S..n.. RP s s - 120V RPS POWER SUPPLY SYSTEM FIGURE N0.5 l
_ __ _ _ __ __ - _ - _ _ _ - - _ _ - _ - - _ _ _ _ _ _
the alternate power supply and only when normal MG set power supply is not connected.
Each power supply to the RPS buses is protected by two electrical protection assemblies (EPA) in series.
These EPA's provide overvoltage, undervoltage, and underfrequency protection to the buses and connected equipment.
Each 120 V RPS bus supplies power to the associated RPS Trip System, Power Range Neutron Monitoring System, and Offgas and Steam Line Radiation Monitors.
The A-RPS Trip System bus (Panel 915) supplies the power requirements of the A-RPS Analog Trip Cabinets (C2228-Al and C2228-A2), Channels Al and A2 Sensors, Channels A1, A2, and A3 Scram, System A Scram Reset, System A Pilot Scram Solenoid Valves, and PCIS Trip Logic Channels Al and A2.
Similarly the B-RPS Trip System bus (Panel 917) supplies System B power , i requirements, including B-RPS Analog Trip Cabinets (C2229-81 and C2229-82) and PCIS Trip Logic Channels B1 and B2.
The PCIS, Group III isolation will isolate the shutdown cooling mode of the l RHR system for any one of the following conditions: (1) Reactor low water level, (2) High reactor vessel pressure, and (3) High drywell pressure.
There are two 120 Vac RHR Isolation Control circuits, one for channel A and the other for channel B, for isolating the RHR suction valves 1001-47 (outboard, 125 Vdc valve) and 1001-50 (inboard, 480 Vac valve off MCC B20).
Channel A is powered from safeguard 120 Vac panel Y-3 off 480 V MCC B17, and channel B from panel Y-4 off MCC B18. (Figure 6). These panels are supplied power from the diesel generator buses.
As shown in Figure 6, when relay K29 deenergizes valve 50 will get a close signal.
Similarly, when relay K30 deenergizes, valve 47 will get a close signal. Relays K29 and K30 deenergize when relays 5A, 5B, SC, and SD deenergize in a one-out-of-two-twice logic. These relays, which are in the PCIS Trip Logic Channels, will deenergize when relays in the RPS sensor channels sense low reactor water level or high drywell pressure.
Relays K29 and K30 will also deenergize through relays K28 and K50. These K28 and K50 relays are deenergized on high reactor pressure as sensed by pressure switches 261-23A and 261-23B, which open on high reactor pressure. If either relay K28 or K50 deenergizes, both 47 and 50 valves will close.
All these relays will also deenergize on loss of power of the RPS, Y-3, or Y-4 and will close the dc valve 47 and ac valve 50 (once power is returned to MCC B20 via 86).
- - _ - _ _ _ _ _ _ _ _ _ _ _ - _ _ _
_.
. _ _ - _ _ _. _ _. - -. - _
1 , ., .s
. . ' . ,
5 e s > > "I I , .... a a w .
o m O o o '**$ g M O o . De D" e
9 x n , ' a.
m ~o 2 n* 2e ' I za8 i .S w.
S.e s ?. c ewe . < ,
- "* > O
. CL C C *
- b '.
O.8 .g e eo eO
__, c eo e a 2.-. O* O* wom 3aH 3 e > N O g e C , O mm e M 9 o o.m.
E g > - e-U 'o E.
X. e o r* - - 2 U Q.
( o - ( Q ( L O$r
e.
e .F e =xa . . ~ m n. m u.
D OOw E n u e
I a = E , n.
e O es a
ai - \\- = - - , - . .
.. s - -4 H HHF
p F- {, . Y gC
o . . . y I
- HHF? +
$ . e , _ , i < ,
e, . .
, a e -
. , -4 F--4 Fi H Fi F4-4: 'F HF
.e u = .b I.
i dHF ^v- . .. .
e e M E E E l <* . g mjs ' . y.
R . o8 h o
i
h.
- less:Y i
i = = l ~ . -. .S .
0% ke NO 0% .$ 4 NO >. D , e * l
---__-_____m_ _ _ _.
l % t x .;. >> , \\
( \\ ' To reset"the isolation, the initiating isolatMn signals must & cleared.
~ Power must be available to at least one of thd two RPS buses and to both Y-3 and Y-4.
The reset switch for the 'RHR shutdown cooling isolation valves (switch 16A-533 on panel 905) must then be operated to energize reset relays 16A-K11 and K12.
(To enyrgize relays K11 and K12, power to panel Y-1 from either MCC B10 or BIS muy.t be available.)
\\t8 ^ ', 4.1.2.3 Operations and Failures During Event ' t Y Initially all 4.16 kV buses vre energized from the startup source.
All the unit source breakersies open.
Bot.h diesel generators were operable ! , O . and in standby.
The shutdown transformer was not available due to ' maintenance which included tie-in of the new "b'.ackout" diesel generator.
/ All 480 volt buses and mEor control centers (MCC's) were energized with i the exception of bus B26 for the intake structure. Bus B4 was abnormmaly aligned and supplied from the 62-B4 cross-tie instead of the normal feed from bus A4.
Bus B6 was being supplied from bus Bl.
When the loss of power occurret both of the running Reactor Protection System-Motor Generator (RPS-MG)' sets were de-energized causing a full scpn sMnal (channels A1, A'E, B1 'nd B2).
In addition, a Primary . Cont intent Isolation Signal @F) was generated due to the above channel triph is well as the temporag interruption of power to buses, Y3 and Y4.
The computer shows these trips coming in just before 2:07 a.m. on November 12.
The PCIS caused residual heat. remq' val (RHR) motor operated pump suction valves 47 and 50 to close and t% operating "A" RHR pump to trip.
Bus Y3 and Y4 were then re energized as both EDGs energized their associated buses. An operator was dispatched to energize the "A" RPS bus from bus
B10.
At about 2:21 a.m., the channel "A" trips reset and this allowed l reset of PCIS. Re-energizing one full RPS channel clears the "one out of two, twice" logic. At about 2 30 a.m. the operators atten'pted to restore shutdown cooling. When "A"i lHR pump was started, another PCIS occurred.
The operators did not notice the 2:30:59 comput- }og alarms which ! indicated trips of several channel A2 parameters. s. lese trips were the first indicator of the fuses which blew in the anal Q trip system cabinet.
In all, four fuses blew, two on channel A2 and one a each channel Al and Bl.
The A2 fuses are located in cabinet f222A-A2.
Only the A2 fuse failures caused a plant response since each cabinet has dual power supply feeds via separate fuses off the same circuit.
An operator was sent to the MG set room where he reported that "B" MG set j l was running on its normal supply but at lowered voltage and current.
The ' "B" MG set would have been disconnected from the "B" RPS bus by the electrical protection assembly (EPA) and disconnected from its power supply when power was lost to bus B22. An interriew with the NOS indicated that although ne recognized that the load on "B" EDG was only about 1300 , q kW, half of rated load, he felt that the diesel was overloaded causing reduced voltage and current on the MG set. In resoonse to this perceived ' condition, the NOS dispatched another operator to strip bus B4 and open , ., _. _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _. _
.p q <g4 .
~' - \\ y l f ! ,; a
- 7
"f /, 1' ! p ypl [
. i j , .the feed to. 50 from ' bus se.. Bus B4 1.. e non-safety bus not normally powered from SObut cross-t%I because W? normal A4/B4 feed was out of J service. 5 hwy 1 thereafter,- the NOS ordered "B" RPS bus transferred to j
.l B10, de energizing the "A" RPS bus a ni.' causing another PCIS.
This ! occurred e.t abcut 2:37 a.m. on November 12. ; Once again, a review of the computer alarm logs clearly shows that only the Al trips came in when RPS ' PA" wu de-engized; the, A2 trips remained in due to the un-noticed blown - fuse d With RPS "B" tik. powered from BIO, the channel "B" trips reset, 4 1 'PCIS was' reset and "A'? and "C" RHR pumps restarted to establish shutdown cooling.
In spite of -the confu<icn over thf RPS behavior, the operators ' were able to re-establish shutdown cooling by' 2:45 a.m., about 40 minutes after the trip which under worst case decay heat conditions, would have ., had ne safety effect.
' , , a I It should be noted thn, an long as buses Y3 and Y4 are energized (i.e.
h both diesels running); fit only takes one' half of. the RPS.to reset PCIS.
J, The ' blown fuses werr ~ not found until 5:55 a.m.
on Novenbar 12.
This ,,
TN scenar.
'comes moreJ significant if the blown fuses occur simultaneously ' _ with a mesel f ailure on the opposite train.
In this instarce with no power to Y4, the shutdown cooling isolation valves would have to be-O >
electrically jurse ed to open them.
The "B" diesel was available
until 11:30 a.m, that morning.
4.s?. 2J * Conclusicais mWRecommendations ,. , ,c ( There was confusion in the control room concerning the loilding on the "B"- diesel genentor and on what actions should be taken to restore power to
- he'RPS bus s The operators did not notice the fuses blowing on channel A even though they were annunciated on th4. comp' uter.
This failure to-notice or, acknowledge computer alarms was also evidenced in the 345 kV system at 1:27 A.M. on November 12 when the line 342 low voltage alarm cams iri on the computer.
It appears that operators are not routinely analyzing; computer alarm data ta ~ assist in assessing plant status.
'A revfew of the operator actions.
should be muie to assess the appropriateness of operator actMns and their analysis of plant data for-y_ this event,
l The 10 ano fuses that blew in the Analog Trfp Cabinets - one in C2228-A1, r-two -in C2228-A2 and one in C2229-B1 - were apparently due to a common _ cause connected with either the starting of the A-RHR pump or the ( switching of the RPS power supplies.
The licensee is investigating the ' roct-causes of the blown fuses.
Had the other fuse in C2229-B1 also . blown, operators would not have been able te reset kHR shutdown cooling isolation.
Corrective. actions based or the results of the investigation should be initiatec; and completed before reactor restart.
During the event, when the B Diesel Generator was out of service and panel ( Y-4 was without pow r, the licensee initiated a temporary modification ' scheme to provide poser to the control circuit of.the RHR suction outboard t . . I h ? c , fy' ! ', f " ' _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -
. . . _ _ _ _ _ _. , _ _ _
l i I
valve 47.
Contingency procedures for the single failure of either onsite emergency power system train (i.e., loss of either Y-3 or Y-4), should be l - considered.
In reviewing procedure 2.4.25, Loss of Shutdown Cooling it was noted that the immediate operator actions provided no specific guidance on action required to restore shutdown cooling.
Similarly, the subsequent operator actions did not specify any mitigating actions to be taken for conditions other than full buses A5 and A6 and PCIS logics available.
The adequacy of this procedure should be reviewed relative to the LOOP event.
4.2 Emergency Diesel Generators 4.2.1 System Description Each emergency diesel generator (EDG) unit is an 18 cylinder, 3485 hp, 900 rpm engine powering a 2600 kW, 4.16 kV, 400 amp, 60 hz, 3 phase generator.
They are located in separate rooms of the diesel generator building which is north of the reactor building.
They may be operated at either the local control station or remotely from the main control room.
In a loss of power situation, they start automatically and energize their associated buses, A5 and A6, within 10 seconds of the time the following conditions > are met: - startup transformer supply breaker open, startup transformer secondary voltage is low and unit auxiliary transformer breaker racked out, or - unit auxiliary transformer breaker is open and the startup i transformer breaker is racked out.
< ) The EDG will automatically start, but not assume loads unless electric ] power is lost, upon receipt of low-low reactor water level or high drywell pressure signal.
The EDG lube oil system is provided with a 480 Vac pre-lubrication (pre-lube) pump which circulates lube oil through a 480 Vac electric heater and then though the normal lube oil flow path. This keeps the engine warm and properly lubricated in order to be ready for an automatic start.
Both the prelube pump and lube oil heater re supplied from emergency 480 Vac motor control centers, B17 and B18 for "A" and "B" EDGs, respectively.
Heater failure is annunciated locally on the control panel when oil temperature drops below 70 F.
Pump failure is similarly annunciated when pressure drops below 2 psig. On an engine start, the prelube pump stops automatically when the engine driven jacket water pump raises pressure above 10 psig. The prelube pump will automatically start when pressure drops below 10 psig.
The heater is interlocked with the prelube pump to prevent overheating if prelube forced circulation is not available.
The I
'
_ _ - _ - _ _ _ J
heater temperature is controlled by a thermostat switch located on the heater assembly.
Lube oil temperature and pressure gages are located locally on the control panels, one panel per engine.
) Instrumentation for diesel generator output current is provided through a current transformer.
The current transformer uses a 600/5 turns ratio; , this circuit is shown in Figure No. 7, and is comprised of an overcurrent I relay and ammeter for each of the output phases of the diesel generator.
'.The current 'from phases A and C are used in a kilowatt hour (KWH) meter.
All circuit components, except for KWH meter, are located in the 4.16 kV switchgear compartment associated with the respective diesel generator.
The KWH meter readings 'are recorded as part of surveillance testing activities.
4.2.2 Operations and Failures During the Event 4.2.2.1 Current Transformer l At approximately 10:00 a.m. on November 12, 1987, an I&C Supervisor in the-vicinity of the "B" EDG output breaker compartment in the A6 Bus switch-gear room noted a zero current. reading on the phase C ammeter.
This condition was reported to the control room, and ' the operations department issued MR 87-730 to investigate the observation.
By 10:45 a.m., the licensee's Electric Lab Engineers were troubleshooting the circuit.
From amp probe measurements taken at the A609 breaker compartment in the lower switchgear room, they determined that current was neither flowing through the ammeter nor the overcurrent trip relay of the phase C ci rc ui t. Verification was made using other switchgear and 480V load center ammeters that the EDG was not operating with unbalanced loading on its phases.
Subsequent to the field troubleshooting, the Electric Lab Engineers observed the operation of the KWH meter for both EDGs, which are located on Control Room panel C-8.
It was observed that the "B" EDG KWH meter was running at half the speed of the "A" EDG, yet both EDGs were evenly loaded. At this point they concluded that the phase C current transformer circuit for the EDG was either open or short circuited.
The trouble-shooting and possible failure modes were reviewed with the cognizant NED Engineer during a call from the control room by the Electric Lab Engineers. The NED engineer concurred with the findings and supported the Electric Lab Engineers recommendation for removing the EDG from service and isolating the failure.
Because the possibility existed that the current transformer was operating with its secondary side open circuited, which is a potentially dangerous condition that involves the development of a very high voltage in the secondary circuit, the Electric Lab Engineers informed the NWE of their recommendation for securing the "B" EDG at 11:20 a.m.
The EDG was secured from service at 11:35 a.m.
and isolation of the unit completed at 1:30 p.m.
to support additional troubleshooting.
Further investigation determined that a wire was disconnected from its terminal lug at the phase C KWH meter output terminal (No. 8), which is shown in Figure No. 7.
The l _-__-____-_--_-_--_--_D
- - _ _ - _ - _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ . A.l4 kV Bus as G&A 47HGn'7" l ,... ' _.
. - IH l " l jw/s. < C - , u, o V) G enenp ron eu w r 84WA WG C ' 1rtr9r
- 4 i
lt<)tp C o #A g N i~
I j %g-- & 7*4'nds feen c it s a , h it
. - J% -+ - . CeNTel-Reap N,
" '&. , PA NE t-C8 , b & .
L e.
vE , e. t o "._g .-== F . ' RR e EE TV "e O 9 F t - "- A i X F t R eE nY - - eR _ n gl-O 'tIg i _ I stuN s A n5
,3 , R E0 - Y8 - x',gO n E _ Y R-F . R R , r OK t r O , U N E A F t T sRiO EAL R S - a l C - I s t , R " ,. a a % ' E Qr- - ct $ T - o mc C - a a a , . E " M A5
- ns L
j e eo g t n R 7$0 + a > E i k i\\ ~ ' ' T. I + t ,P ir ,E ^^ < R h v ,Y , ( J ,1 I ,R ,D A / j l ,M , ,l " D ,A r -- E 1 . M ~~- q a ~ S l f X S E e E R U H R G r n0 tn I P m ',y @s,
- ,
u F o v4 C, - - M I f - k 3 C p, .ge"r" - - O pi e 8
- I
R e t -.r C ,l- , ] E gf 1 . ee s t . s t - pt . ? X F . . ' ' . - -- d
- -
',.. - ._ X ,. y WW Wy gc. .- PO '*. n' 7-g ' k s ) .- .R y V - C - g C I
- E
- C I R G 8 A o -- " t d 4 4 0 0 g 1 . > - - g C V Y ( MM ) p e, AV , t t , { y c c '. , t e e N e, - . m o o t t m , R o - - - , .- E g ,j a n , g T e e . . r ( s s . , $, e- ' A e e . g l = a g ' .v I e e g - H s S - t ( , a n T,d j - O S. S , t SO R , n I R O Et t r ' ii: l o R R S ,R C s E t i E , I S , rN A T. C t E , _ uO s xC E N , 4 s E t e n ij" O C a , E - r e N s E , E n , f A r , C e C t g oje uu C mn[s g t i M u c s o P s l u o , C T 'a , C l w e? s __-_ Component Description There are two main screw-type compressors (K-110 and K-111). One main compressor can normally supply the entire station load. The compressors are cooled by turbine building closed cooling water (TBCCW). Air com-pressors K-110 and K-111 are powered from 480 V load centers B-8 and B-5, respectively. The maximum working pressure for K-110 and 111 is 152 psig and the normal working pressure is 102 psig. At 102 psig the compressor capacity is 655 cfm. The screw compressors have an internal outlet check valve and a check valve in the discharge line leading to the receiver tank inlet. The check valves prevent system pressure loss by compressor leakage. There are three smaller air compressors (K-104A, B, C) which are vertical, single-stage, double-acting reciprocating compressors, rated to deliver 160 scfm at 105 psig (see Figure 9). Air compressors K-104A, K-1048, and K-104C receive power from 480 V, MCCs B-15, B-14, and B-10, respectively. A normally isolated pressure reducing valve in the cross-over line, PCV 4370, is installed between the high pressure distribution header (up-stream of the air dryer) and the low pressure distribution header. The cross-over can supply low pressure service air if the low pressure blowers fail. Some systems which use instrument air as the motive force for a com-ponent's operation, have a check valve and an accumulator in their supply. This ensures that the components will continue to operate even if HP air header pressure is lost. Seismic class I accumulators, associated piping, and check valves are provided for the following equipment: Torus to secondary containment vacuum breaker butterfly valve
Main steam isolation valves
Main steam relief valves
Service air header valve A0-4353 isolates the high pressure service air to low pressure service air header cross connect at 85 psig (decreasing) if a problem occurs in the high pressure air system. This ensures that vital services in the high pressure air header will receive all available air. Valve A0-4350 isolates the high pressure service air header, in case of a leak, when the upsteam pressure switch senses 85 psig (decreasing). Valve A0-4365 is shut to isolate the non-essential instrument air header, in case of a leak, when the upsteam pressure switch senses 80 psig decreas-ing. The detailed system interrelationships and effects of losses of the various high pressure service and instrument air headers are presented in Appendix F. 4.3.2 Operations and Failures During Event Operating Procedures for Air Systems Instructions for operating personnel to perform normal operation of the instrument and service air systems are provided respectively, in PNPS _________________a ]l Il l i ^- A ' O'$g _E
? t f/ te ef R m E t v y-N l y E tE - ' C C E-l R E N R _ . 2}v N g j N g-N . . O t / a < EE EV L A R-t= N , V R - v Y O t ~ t T T T C ,E E A N E E ,V' F R M ,A",. L A E . A ,V . P E M . E - t. S " i E A R M "- - G 6= . N . S M A W C A . f R R fA E g R Mf E t E Of A A N T T N U h S E O A G C N E ' = I
- =E"
@' E P . I A R O P N R M T' E ' O t A R E W" S 't R E S d O U E E R "c G n S P / dia, ' E I M S F O E C l _ R en E o P r f - / ( E M Y A T.- n p / g @@Kc O ,
W i C A S C S E = P h E E - n L O ] ' # e L A N .M S '- C
- -
R ' \\ C E I - P w E T Ae l FI r-M Y L RO T n
e D O
O M E R E , t H R , T ,E E E , , t A f t R , E i (=A
t , u W F , O e R e I
A D g
O O H N Atg \\%' hsQKh7 x _. U , , e - , - - t g8 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - _ _ _ - _ _ _ _ _ _ _ _ _ _ _ Nuclear Operations Department (N00) procedures No. 2.2.36, " Instrument Air System, " Revision 15, dated October 23, 1987 and No. 2.3.37, " Service Air System," Revision 9, dated July 3, 1987. Those procedures appear to provide an appropriate level of detailed and clearly written instructions to assure that operating personnel conduct proper system lineup and component operations for an operating plant condition. No. instructions, however are included in those procedures for abnormal operation of the air systems. Operator actions for emergency operation of the instrument and service. air systems are provided in PNPS NOD procedure No. 5.3.8, " Loss of Instrument Air", Revision 10, dated October 30, 1987. Procedure No. 2.2.36, " Instrument Ai r Systems", is written for a normal operating plant condition during which all five air compressors would be expected to be operable. [As noted in the procedure, however, there are no technical specification limiting conditions for operation or admini-strative limits regarding instrument air system components.] One main air compressor, K-110 or K-111, is capable of normally carrying the station's entire load while -in the leading position control. The other screw com-pressor will be in the lagging position control. The leading compressor cuts in at 102 psig (decreasing) and cuts out at 108 psig (increasing), whereas the lagging compressor cuts in at 93 psig and cuts out at 103 - psig. Two of the smaller units, K-104A, B or C, are normally in the " standby" position control and will either cycle one at a time or together dependent on system pressure demand. The other smaller compressor is normally in the "off" position. Air System Configuration and Status on November 12, 1987 On November 12, 1987, prior to the loss of oftsite power (LOOP), the instrument air systems were not in a normal operability configuration, as shown below. Instrument Air Systems Operability Status Prior to LOOP Normal Power Emergency Power Operability Component Source Source Status ] Compressors K-110 B-8 None On - Lagging, tagged for known oil leak K-111 B-5 None On - Leading K-104A B-15 "A" EDG Tagged out of service for compressor replacement. ] l _ _ _.. _.. _ _.. _ _. _ . _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _. _ _ _ _____J - - _ - _ _ _ - _ _ _ _. K-104B B-14 "B" EDG Control switch-off not tagged, but known head gasket air leakage. K-104C B-10 "A" EDG Tagged out of service for: (1) TBCCW (aftercooler) air leak (2) electrical ground (3) oil leak (4) failed capacity test. Air Dryers X-105 B-10 "A" EDG On X-160 B-23 None On - not cycling properly. Losses and Restorations of Instrument Air System When the LOOP occurred at 2:06 a.m. on November 12, 1987, on-site power was lost to the screw compressors K-110 and K-111. Although both the "A" and "B" emergency diesel generators started as a result of the LOOP, none of the reciprocating compressors started as two were tagged out of service and the control switch for the third was turned off. Thus, with no operating compressors, a total loss of instrument air occurred. The resultant reduction in air header pressure caused block valve isolations to occur within five to ten minutes after the LOOP. Air header pressure continued to drop off to about zero psig in the next fifteen to twenty minutes. At 3:10 a.m. compressor K-104B was started at the local control panel, and the essential instrument air header slowly repressurized to 40-50 psig. The air system remained in that conditien until 11:30 a.m. when the "B" EDG was secured. That action caused compressor K-104B to stop upon loss of its emergency power source, and air system pressure again depressurized
to zero psig in 20-30 minutes. The air system remained depressurized ' until af ter of f-site power was restored and. screw compressor K-111 was i re-energized at 11:45 p.m. The instrument air system achieved a normal l 102-108 psig operating pressure band at 12:05 a.m. on November 13, 1987. Except as a result of air compressor inoperability at the time of the loss of off-site power or loss of the normal and emergency power sources to the l air compressors described above, no other failures occurred to instrument air system components during the total or partial losses of the compressors. All system header isolations and system interactions occurred as expected for the cold shutdown condition of the plant. - - - _ -__ ______-___ _ _-_-___-_____--_-_ - __-_ - _ _ _ _ _ . _ _ _ _ _ _ _ _ _ ._ - _ _- _ _ _. _ _ _ _ _ _ _ 4.3.3 - Air System Recovery Operator Response Procedure No. 5.3.8, " Loss of Instrument Air," is written for the expected plant response to a degraded or loss of instrument air header pressure that occurs during a normal operating plant condition. The procedure identifies automatic and operator actions with respect to feedwater, condensate, control rod and cooling water system operation. The procedure specifies that subsequent operator action should include (1) the initiation of efforts to re-establish normal instrument air pressure, and (2) the isolation of makeup to cooling water systems surge tanks. Both of. . those actions were taken by plant operators af ter the loss of off-site power occurred. Within one hour of the event initiation, operators verified that proper air system isolations had occurred, manually isolated makeup to the cooling water surge tanks and started compressor K-104B, the only available air compressor. When air header pressure was restored to 40-50 l psig, plant operators checked that air system component configurations
were unchanged. For the remainder of that shift, plant operators maintained the instrument air system in a degraded, single-compressor, i mode of operation, while periodically checking the status of air system ! conditions during plant tours. The air system status was also confirmed by the relieving day-shift operations personnel soon after turnover of the shift watch stations. Emergency Procedure Adequacy / Procedure No. 5.3.8, " Loss of Instrument Air," contains the essential operator actions to place the plant in a stable condition, in the event that the instrument air system becomes degraded or inoperable while the plant is operating. However, both licensee and AIT personnel noted, during the review of the LOOP event, that the procedure does not address subsequent operator actions with specificity. For example, the procedure does not address actions to be taken when the essential air system header remains in a partially pressurized, as compared with a fully depressurized, condition. Also, the procedure does not provide instructions for operators to verify specific component response functions have occurred upon loss of instrument air. Licensee management has directed that procedure No. 5.3.8 be reviewed for appropriate changes, as , a result of the lessons learned from the LOOP event. ' Air System Restoration Actions At 7:30 a.m. the nuclear operations department manager directed the systems group to pursue options for restoring air system pressure, and a parallel path recovery plan was initiated. An unplanned priority "A" maintenance request was promptly written to investigate and repair a suspected ground to the motor of compressor K-104C. At the same time, - _ _ _ _ _ . _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - __, - _ - _ _ - - _ _ _ _ -. _ _ _ -. _ _ -_ _ _____ ____-_ _ _ _-______-_________-_ -__ __ - __ systems engineers determined that there were no suitable portable air compressors onsite that could be connected temporarily to the instrument air system, and a vendor (Atlas-Copco) was contacted to locate an oil-free compressor. Meanwhile, systems engineers developed a temporary modifica-tion for installation and use of an offsite air compressor. These efforts continued throughout the day. A portable compressor was obtained, installed and tested by 10:30 p.m., but was kept in a standby mode pending the near-term restoration of off-site power. After power was returned to in plant buses, compressor K-111 was re-energized at 11:45 p.m., and the instrument air header attained normal operating pressure shortly thereafter, as discussed in Section 4.3.2. Subsequently, after completion of the corrective maintenance for compressor K-104C, it was satisfactorily tested at 1:30 a.m. on November 13, 1987. The recovery actions taken for restoration of the instrument air system are summarized in Appendix G and discussed further in the sections that follow. Plant Operation with Degraded Instrument Air System , As indicated in the sequence of events, between 9:30 a.m. and 10:00 a.m. on November 12, 1987 the senior system specialist (system engineer) for air systems completed an initial walkdown of the instrument air system to determine proper isolation header and component response functions. Based on his knowledge of system design features, the engineer recommended to on-shift operations supervision that compressor K-1048 be stopped in order to fully depressurize the essential instrument air Leader. The basis for that recommendation was that it would be better to have the air system in a condition that had been previously evaluated as to the system response and consequences, as opposed to maintaining the header partially pres-surized at 40-50 psig. Operations section management was still consider-ing this recommendation and investigating alternate methods of restoring the instrument air system wt en compressor K-104B was stopped as a result of manual shutdown of the "B" emergency diesel generator. The appropriateness of operating the instrument air system in a degraded, partially pressurized condition or in a fully depressurized condition is being assessed by the licensee in conjunction with the review of procedure No. 5.3.8, " Loss of Instrument Air," as discussed above. Compressor K-104C Maintenance Activities Based on their review of outstanding maintenance requests and tags for reciprocating air compressor K-104C, operations and maintenance personnel attributed the electrical problem to a motor ground. The priority "A" < maintenance request (MR) No. 87-827 was originated to investigate and repair the suspected ground, using PNPS nuclear operations department procedure No. 3.M.3-8, "In;pection/ Troubleshooting - Electrical Circuits". The licensee's initial investigation determined that the fault was ) actually within the compressor's circuit breaker in motor control center j (MCC) No. B-10,. compartment No. B1064, and further work was conducted I _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ ' ' - - ' - - - - - - - _ _ _ _ _. _ _ _ _ ! ) J using procedure No. 8.Q.3-3, "480 Vac Motor Control Center Testing and Maintenance." The electrical fault probable cause was then identified as the A phase motor terminal wire that had lost contact, due to only a partial. number of ] strands that were landed in the compression terminal lug which in turn caused the wire to overheat and burn away. The wires were restripped and ' terminated on a new terminal block. As noted on the procedure No. 8.Q.3-3, Attachment B, "480 Vac Motor Control Center Inspection and Maintenance," completed November 12, 1987, breaker B1064 had been reworkr3 during refueling outage No. 7. The A phase field wire to the ! motor apparently did not have enough strands wired in the compression
terminal. The wire then burned frec and during the process heat destroyed the terminal. Therefore, the terminal board was replaced. No degradation was noted to other components, except slight discoloration of the field wire. While conducting the MCC maintenance post-work testing, compressor K-104C starting and running currents were obtained, but the compressor was not loaded because an oil leak was discovered on the lower crank case side cover. Priority "A" maintenance request No. 87-733 was then initiated by the nuclear watch engineer to change the oil pump gasket, check the oil pump relief valve and change the oil filter using procedure No. 3.M.1-11. During subsequent repair activities maintenance and system ' engineering personnel determined that the oil leak emanated from the oil pump itself, which was also replaced. In addition, a new cover gasket was fabricated. Post-maintenance testing was completed satisfactorily late on November 12, 1987. Compressor K-104C loaded and unloaded within the design criteria, and there were no oil leaks at 30 psig oil pressure. The post-maintenance testing was completed satisfactorily at 1:30 a.m. on November 13, 1987, and compressor K-104C was declared returned to normal at 10:40 a.m., after all tags /isolatiors were removed and MR logs and files were updated. The maintenance activities related to compressor K-104C were conducted methodically and properly, in accordance with approved maintenance procedures. There was close coordination between maintenance and operations personnel of the various repair and post-maintenance testing activities, as well as continued oversight by the senior system specialist (system engineer) of the comp re <., so r MCC breaker and oil pump repairs. Subsequent to completion of the compressor maintenance, on November 16, 1987 maintenance group engineering initiated PNPS failure and malfunction (F&M) report No. 87-641 regarding the electrical failure of breaker B1064 as described above. The major concern identified on F&M No. 87-641 was that compressor K-104C was not operable when required. Further, that deficiency was identified on August 3, 1987, by nuclear watch engineer tag No. 46-227. The licensee's F&M No. 87-641 review, including determination of a requirement for root cause analysis and corrective action plan, was not yet completed at the end of the AIT inspection. The licensee's further actions regarding F&M No. 87-641 will be reviewed during a
subsequent NRC resident inspection. ! ! _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ , _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ Off-site Air Compressor Installation About 8:00 a.m. on November 12, 1987 station and engineering management - determined that there was no oil-free portable air compressor onsite that would, be acceptable for temporary installation into the instrument air system. Therefore, at 8:15 a.m. a compressor vendor, Atlas-Copco, was contacted to locate a suitable oil-free compressor that could be promptly . delivered to PNPS. The vendor's location and delivery of an off-site air compressor is summarized in Appendix G. ' Temporary modification- (TM) No. 87-30, " Temporary Instrument and Service i Air Supply", was initiated to provide a properly approved installation of the portable back-up supply of compressed air to the high pressure service air and instrument air systems to be used if both the K-110 and K-111 compressors are unavailable. Specifically, TM No. 87-30 provided for installation of a temporary high pressure compressed air supply to the outlet piping of compressor K-104 A. The TM No 87-30 was similar to TM No. 87-21, which had been approved for use during the recent months of the plant shutdown, while compressors K-110 and K-111 were unavailable for scheduled maintenance. The TM 87-30 was properly approved both by the onsite review committee. (ORC) during ORC Meeting No. 87-133 and by the nuclear operations department manager on November 12, 1987. The nuclear watch engineer subsequently authorized installation of the temporary modification, using MR No.87-736. The TM No. 87-30 installation was completed as authorized and the back-up Atlas-Copco air compressor was operationally tested on the evening of November 12, 1987. However, pending imminent restoration of off-site power and the availability of compressor K-111, the backup compressor was maintained in a standby mode and was not used to pressurize the instrument air system. 4.3.4 Changes and Improvements Recent organizational changes and an air system status review completed in October 1987 contributed to the licensee's ability to promptly implement sound recovery actions in response to the various losses of the instrument air system during the loss of offsite power event. In early 1987, the systems group was formed as part of the technical section within the nuclear operations department of the PNPS restart organization. As discussed in the PNPS Restart Plan, Volume 1, transmitted by BECo letter to the NRC dated July 30, 1987, the establishment of the systems group was patterned on INPO Good Practice TS-413, "Use of Systems Engineers". The systems group is to provide greater depth and continuity in the oversight of major plant operating systems at PNPS by assigning " ownership" of each system to a qualified individual. These personnel are also assisting in the restart effort by reviewing and verifying work completion in their respective areas of responsibility. _ _ _ _ _ _ _ _ _ _ _ _ _ _ , _ _ - _ - _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ _ _ , ! l In September 1987, the senior system specialist position for air systems l and certain other balance of plant mechanical systems, e.g., auxiliary-I off gas, demineralizers, etc., was filled by a contractor engineer. That .) individual was tasked with reviewing the overall. status of the material { condition of the " owned" systems and to recemmend actions to improve overall system operation and reliability. With respect to the PNPS I instrument and service air systems, a comprehensive study was conducted as documented in an office memorandum SG-87-384, from D. Rosinski to W. Clancy dated October 27, 1987, " Air System Status. Report." The PNPS study was conducted as part of the licensee's review of NRC IE Notice 87-28 " Air Systems Problems at U.S. Light Water Reactors," which attached i NRC case study report AE00/C701, " Air Systems Problems at U.S. LWR's." l The NRC cace study report reviewed a large number of problems experienced ! by U. S. light water reactors due to air system related failures or system l degradations, identified eight root cause deficiencies and included five ! recommended actions', ' The PNPS study, which included review of applicable current industry and
regulatory standards for air systems,- vendor technical manuals and i existing maintenance and surveillance procedures, addressed three of the eight NRC case study root cause deficiencies including: (1) mismatched equipment-the air quality capability of the installed filters and dryers do not match the design requirements of the equipment using the air; (2) ! preventive maintenance - system maintenance is not always performed in accordance with manufacturer's recommendations; and (3) air quality - air quality is not monitored periodically to assure that system dryers and filters are working properly. The PNPS air system status report identi-fied the specific PNPS status and included specific recommendations with respect to the three root cause deficiencies described above. Overall, the report concluded that the PNPS air systems are in poor material condi-tion and the preventive maintenance program does not meet the current NRC recommendations. Corrective actions to implement the suggested NRC guide-lines and improve overall system performance were divided into four short-term and seven long-term items, which collectively represent a j substantial commitment to improving the material condition of the air j systems. i The licensee's actions taken in response to IE Notice 87-28 are being reviewed separately as part of the ongoing NRC resident inspection activities, and the delineation of specific recommended actions and their schedule, as addressed in the PNPS status report, is not considered within the scope of the AIT report. However, the team again notes that the i licensee's study was comprehensive and detailed and included sound recom-j mendations. Also, as a result of the system specialist's extensive { involvement in conducting the study and developing the report, substantial system knowledge was acquired that contributed to the overall licensee recovery actions to the losses of the instrument air system on November 12, 1987. - - - - - - - - - - - - - - - - - - - - - _ _ _ _ _ _- i l Based on the PNPS systems group status report recommended actions to improve overall air system operation and reliability, and in specific consideration of certain lessons learned during the loss of offsite power event, on November 16, 1987 the systems group initiated an engiraering evaluation request (EER), " Station Air Systems Deficiencies." ine EER highlighted in summary that the instrument and service air systems require upgrading and presented a detailed description of typical problem areas and recommendations that require further corporate nuclear engineering evaluation. The problem areas and recommendations addressed four short-term (pre-startup) items regarding (1) the inadequacy of the existing temporary system (TM 87-30) for connecting a portable air compressor to the station air system; (2) the existing design adequacy of air header. . i isolation (block). valves upon loss of air pressure and subsequent restoration, (3) air dryer and filter sizing for current system loading and operational transients, and (4) clarification of air quality specifi-cations and testing requirements. Six long-term (post-startup) items were identified for engineering evaluation including: (1) use of temperature control. valves in the compressor cooling system, (2) installation of a central base / trim load station to provide centralized control and system status, (3) adequacy of having air receivers placed between the compres-sors and air dryers, (4) non corroding intake piping installation, (5) the need for the installation of d/p gages across filters and air dryers to monitor operation and determine need for replacement, and (6) the need for an in-line dewpoint sensor. In addition, the EER identified the current priority of the 40 identified work items related to the air systems, 30 of which are considered priority one (restart). The EER was in the onsite management review and approval process at the time of review by the AIT team, and was subject to in process revision prior to approval by the nuclear operations department manager. Nonetheless, the EER demonstrated the systems group awareness of the importance of correcting known or perceived station air system deficiencies. 4.3.5 Conclusions and Recommendations Initial and subsequent operator actions in response to the losses or degradation of the instrument and service air systems, as a result of the loss of off-site power event on November 12, 1987, were properly conducted and were in accordance with an approved facility emergency procedure. Although procedure No. 5.3.8, " Loss of Instrument Air", was not considered deficient for the events that occurred with the plant in a cold shutdown condition for an extended period, the AIT team members noted that the l procedure lacks specificity of subsequent operator actions. Other l improvements or changes may be warranted, based on a planned licensee ' review of the procedure for its overall adequacy for performance capabi-lity while the plant is in a normal operating condition, and based on i i ricorporation of appropriate revisions to reflect air system design I changes or modifications. I i - - . - _ _o _ _ _ _ - _ _ 1 ( i The instrument air system non-availability that occurred on November 12, 1987, never threatened the ability to maintain the reactor in a safe, controlled cold shutdown concition. Further, air system header isolations and component actuations occurred as designed, as verified by plant operators and system engineers. The extended loss of the instrument air system became a nuisance, but not a detriment, to operator performance l during the loss of off-site power condition, as compensatory measures were required to monitor certain tank or water bay levels or to monitor and pump various plant sumps. ! Recovery actions for restoring air system pressure were appropriately cautious and methodical, with proper consideration of the stable, non-challenging reactor and plant conditions. Operators started the only available air compressor within one hour of the loss of of f-site power. Day shift recovery actions benefited from the systems engineer's knowledge of air system design and material condition. Operations, maintenance, engineer and other support activities were well integrated into the over-all recovery ef forts. Sound management decisions for recovery actions were based on well understood plant conditions, operational needs and air system problems. Contingency plans and schedules considered realistic and periodically updated time estimates fur equipment maintenance, delivery, temporary modification and testing. Extensive air system deficiencies had been identified during a recent , comprehensive review by the systems group. The review concluded that the i PNPS air systems are in poor material condition and the current preventive j maintenance program does not meet the current NRC recommendations. How-j ever, substantive recommendations were made to escalate the priorities of j previously identified and additional maintenance activities and to conduct i engineering evaluations of identified problems. These activities are being conducted in support of the utility management assessment of restart readiness. Licensee efforts to upgrade the existing design and material condition of the instrument and service air systems, and to improve the loss of instru-I ment air procedure should continue as planned. No additional NRC recom-mendations are considered necessary as a result of the AIT review and assessment of the November 12, 1987 loss of off-site power event. 5.0 Human Performance 5.1 Organization l l The organization established by Boston Edison Company for the normal ] operation of Pilgrim is divided into the Nuclear Operation Organization j and the Nuclear Engineering and Quality Assurance Organization. The ' Nuclear Operation Organization directs operational activities at Pilgrim while the Nuclear Engineering and Quality Assurance Organization provides i onsite and offsite support. Both of these organizations report to the Senior Vice President, Nuclear who reports directly to the Chief Executive Officer. ! j ! l _ _ _ _._ _- -- .- . _ _, ! In addition to normal operations, these organizations are staffed and qualified to take those emergency actions necessary to implement the Emergency Plan including immediate protective measures. Pilgrim Station is operated by the Nuclear Operations Department (NOD) of the Nuclear Operations Orgainzation. The NOD Manager serves as the Station Manager and, as such, is responsible for the overall safe opera-tion of Pilgrim. The NOD Manager reports to the Vice President-Nuclear Operations. (The position of Vice President - Nuclear Operations is temporarily being filled by the Senior Vice President, Nuclear). Within NOD, there are five (5) areas of responsibility. Each area is headed by a Section Manager who reports to the NOD Manager. The Operations Section, headed by the Operations Section Manager, is responsible for the safe and proper operation of the plant. The Section Manager's immediate assistant for operational activities is the Chief Operations Engineer (COE). The immediate on-shift position of authority to carry out plant operations responsibility is the Nuclear Watch Engineer (NWE), who oversees all shift operating personnel. The Nuclear Operating Supervisor (NOS) functions as an assistant to the NWE and in the NWE's absence, assumes the responsibilities of the NWE. The Nuclear Plant Operators are responsible for the manipulation of controls as necessary to perform plant operations. The Nuclear Auxiliary Operators are responsible for performing component and/or system operations outside of the Control Room. The Nuclear Plant Operators and the Nuclear Auxiliary Operators are responsible to the NOS and take direction from the NOS or the NWE. Other on-shift operations personnel include a Radiochemistry Technician and a Shift Technical Advisor (STA). The Maintenance Section, headed by the Maintenance Section Manager, is responsible for the maintenance of the plant. At this time, the Main-tenance Section Manager holds the additional responsibility of Outage Manager. Maintenance Section responsibility includes the periodic preven-tive and corrective maintenance of plant systems and equipment, the day-to-day general upkeep of onsite facilities and the procurement of parts and supplies to support these activities. The Maintenance Section provides on-shift maintenance personnel. The Technical Section, headed by the Technical Section Manager, provides engineering services and plant analysis support at the Station. This Section provides many of the resources to staff the Technical Support Center (TSC) in the event of an emergency. The Radiological Section is responsible for the implementation of the Pilgrim radiation protection and radiological / environmental monitoring programs. The Radiological Section provides one or more Radiation Protection Technicians on each shift. The Administration Section is responsible for station personnel health and administrative services. i _ _ _ _. _ - _ _ _ _ _ _ _ _ _ _ _ _ - _- __ __ - __ ___ The Pilgrim Emergency Response Organization is composed of pre-designated I station and corporate personnel trained to augment shift operating ) personnel in an emergency. The normal shift operating group provides the ' initial response to an emergency. This group is trained to be self- ! reliant for a sufficient amount of time (30 to 60 minutes) for personnel assigned to the Emergency Response Organization to assemble and integrate i smoothly into the emergency response. ' The positions of Emergency Director and Emergency Coordinator provide the direction and coordination needed for an effective emergency response. The Emergency Director is responsible for the direction of all onsite emergency response activities. The Emergency Coordinator is responsible for the overall coordination of the BECo emergency response effort and for the coordination of that effort with the activities of offsite support agencies. l In all emergencies the NWE is responsible for initiating the implementa-tion of the Emergency Plan. When an emergency situation is initially identified, the NWE will classify the event and assume the role of Emergency Director. An emergency may be declared anytime that, in the judgement of the NWE, the plant status warrants such a declaration. The initial emphasis of the Emergency Director is to assess the emergency situation, make required notifications to offside agencies and call out the Emergency Response Organization, if needed. The NWE will continue as Emergency Director until relieved. The NWE will brief the incoming Emergency Director on the emergency classification, plant status, radiological conditions and protective measures. Once briefed, the incoming Emergency Director will relieve the NWE of all Emergency Director responsibilities. Command and control of the emergency will then transfer to the TSC once that facility has been activated. The Emergency Director reports to the Emergency Coordinator. The Emer-gency Director is normally the Station Manager, the Technical Section Manager, the Operations Section Manager or the Chief Operating Engineer. The Emergency Director is required to be an ANSI 3.1 qualified manager / supervisor or a licensed senior reactor operator. The Emergency Coordinator is normally the Vice President-Nuclear Opera-tions. The Emergency Coordinator is required to be an ANSI 3.1 qualified manager. The TSC Supervisor and staff are responsible for providing technical support to the NWE in mitigating accidents and their consequences. The TSC staff provides the operating staff in the Control Room with in-depth diagnostic and corrective engineering capabilities while relieving them of peripheral duties and communications not directly related to reactor system manipulations. The staff is able to analyze current and projected plant conditions and, through communications with the NWE, provide 50 ) technical support and recommendations regarding emergency actions. The TSC staff is composed of several engineering and technical disciplines such as reactor engineering, plant analysis, operations, chemistry, main-tenance, instruments and controls, electrical engineering and fire l protection. ! 5.2 Event Classification and Notification i The Emergency Plan provides four emergency condition classifications. In ascending order of severity these classifications are Unusual Event, Alert, Site Area Emergency, and General Emergency. The first two classi-fications involve degradation of the safety of the plant and initiate precautionary measures should the situation become worse. For loss of power conditions, an Unusual Event is declared if there is irradiated fuel in the vessel and RCS temperature is greater than 212 F, and (a) there is a loss of all onsite ac power capacity; or (b) there is a loss of all of f site power (i.e. loss of line Nos. 342 and 355, and shutdown trans-former). For loss of power conditions,' an Alert is declared if there is irradiated fuel in the vessel and RCS temperature is greater than 212 F, and (a) there is a loss of all onsite dc power for 15 minutes or less, or (b) there is a loss of all offsite power coincident with loss of both emergency diesel generators. On November 12, 1987, shortly after the LOOP, control room personnel reviewed emergency action levels (EALs) prescribed in the Pilgrim Emergency Plan. Based on their understanding of plant conditions at the time, they concluded that no threshold had been reached requiring actions prescribed by the Emergency Plan. In accordance with the provisions of 10 CFR 50.72 Immediate notification requirements for operating nuclear power reactors, control room personnel notified the NRC Operations Center via the emergency notification system (ENS). This notification was made at about 2:55 a.m. (about 50 minutes after the LOOP). The licensee reported a scram signal on loss of offsite power; 40-50 mph winds with snow; the plant was stable in cold shutdown; and the diesels were manned, loaded and carrying house loads. Licensee personnel onsite made additional notifications and took action to supplement onsite resources as summarized below: - Control room personnel contacted the Nuclear Operations Department Manager at his home at 2:20 a.m. He arrived onsite at about 3:00 a.m. - Between 3:00 a.m. and 4:30 a.m. the Nuclear Operations Department Manager notified the NRC Resident Inspector the Senior Vice President-Nuclear, the BECo Public Information representative, and the Middleboro State Police (local Civil Defense). The Nuclear < Operations Department Manager also attempted to reach the Chief Operating Engineer; however his phone was inoperable due to the storm. -_____-_____-___-_____- - _______ - _ - _ - _ _ _ - =- - . _ _ - - - I l I - At about 2:35 a.m. the Assistant Director of Outage Management who was on shift contacted the Maintenance Section Manager. The Main-tenance Section Manager arrived onsite at about 7:30 a.m. - At about 3:20 a.m. the Senior Electrical Engineer in the Maintenance Section was contacted. He arrived onsite at about 4:00 a.m. Between 4:00 a.m. and 5:00 a.m. the BECo Emergency Preparedness - organization made notifications to Community Representatives for the towns of Marshfield, Duxbury, Kingston, Carver, and Plymouth. Taunton and Bridgewater were notified later in the day. - At about 5:30 a.m. the BECo Emergency Preparedness. organization notified the Commonwealth of Massachusetts, Departments of. Public Health and Civil Defense. Community Representatives and Commonwealth representatives were again notified when the "B" Emergency Diesel Generator was shutdown and again when normal power was restored to Pilgrim. [ Note: The NRC Resident arrived onsite at about 5:00 a.m. He established telephone contact with Regional and Headquarters Manage-ment to apprise them of the situation. Periodic briefings by the resident inspector were continued throughout the event. After some initial telephone difficulties, an open line was maintained between the NRC Operations Center and the Pilgrim Control Room commencing at about 9:00 a.m.] As determined during AIT interviews, licensee personnel re-evaluated the emergency classification of the event at various times during the recovery effort. The Nuclear Operations Department Manager stated that he re-evaluated the emergency classification and EALs af ter he arrived onsite and again when the "B" EDG was taken out of service. The day-shift Nuclear Watch Engineer stated he also re-evaluated the event classifica-tion after the "B" EDG was taken out of service. TSC supervisory personnel also stated they evaluated the event classification. These ! individuals believed that an " Alert" should be declared if the "A" EDG was lost (after the "B" EDG was shutdown). Based on criteria in the Pilgrim Emergency Plan, an Alert would not be required for a complete loss of all offsite power coupled with a loss of both diesel generators unless RCS temperature was above 212 F. 5.3 Use of Support Organizations and Personnel , i The management of support organizations and personnel was examined from ! four aspects. These were 1) immediate support to the operating organiza-tion af ter the occurrence of the LOOP; 2) NED support; 3) use of the Technical Support Center; and 4) use of the Maintenance Sectio _ _ _ _ _ _ _ _ _ _ _ _ The licensee initially supplemented the plant operating organization with additional personnel to deal with the recovery from the LOOP event. Once the plant had been stabilized and initial notifications made, the Station Manager and Senior Electrical Engineer were recalled to the site. These two individuals arrived at about 3:00 a.m. and 4:00 a.m., respectively. Recall of other resources was not undertaken by the Station Manager or the NWE. In light of the plant conditions, the severe weather, and the impending arrival of the daytime work force this decision appears to be appropriate. The NED support was on an informal basis until the Technical Support Center was staffed at about 1:00 p.m. Early in the recovery effort, Prior to staffing the TSC, it appears that NED management viewed the event as essentially a plant operational matter. Interviews by the AIT indicated that numerous staff-level conversations occurred between NED personnel and onsite system engineers, technical personnel and various Maintenance Section individuals. Most of these conversations concerned electrical engineering issues. One of the significant informal contacts between the NED and the site concerned the "B" Emergency Diesel Generator (EDG). During the morning of November 12, an electrical engineer in the Power Systems Group discussed the apparent failure of the "C" phase ammeter with onsite engineers from the Electric Lab. This type of discussion is typical of routine technical advisory discussions between NED and the site, according to the NED Manager. In this instance, NED management was not involved in discussions that led to the shutdown of the "B" EDG. At about 1:15 p.m., NED established an open line to the Technical Support Center (TSC). By 2:15 p.m., the Power Systems Group established 24-hour coverage to support TSC requests, and a representative of NED was onsite in the TSC. From this time until the recovery effort was completed, NED provided constructive support to the station via the TSC. Activities included safety evaluations of temporary procedures and assessments of several temporary modifications being considered by the site. One of these modifications, which was not implemented. in part on NED recommenda-tion, related to providing vital power to one of the non-vital air com-pressors. A major support activity involved the " administrative staffing" of the TSC. The TSC was not formally activated pursuant to the Pilgrim Emergency Plan; however at about 12:45 p.m., the station manager directed the technical section manager to administratively staff the TSC in support of the'NWE. Several individuals had already been working on an ad-hoc basis in the TSC reviewing plant conditions. The TSC was staffed by about a dozen individuals and logs were established on November 12, 1987 at 1:08 p.m.. The positions of Emergency Director and Technical Support Center Supervisor were assumed by the Technical Section Manager and Technical Group Chief Engineer, respectively. No offsite aspects of the Emergency Plan were deemed necessary. Operations - ____-_-_- __ Center " administrative support" was provided by the Maintenance Section Manager and Maintenance Section personnel. The TSC remained staffed until 3:45 a.m. on November 14, 1987. The TSC carried out a number of actions in support of the NWE. These actions included: - Monitoring and tours - of the plant to ascertain sump levels; moni-toring of water accumulation and subsequent observation of the decontamination of the corridor area on the - 1 foot level of the Process Building (See Section 5.7). - Monitoring of fire protection system components to assure that no degradation was caused by the loss of station air. Review and approval of various temporary procedures and temporary - modifications. These actions included Operations Review Committee .(0RC) reviews required by Technical Specifications. Monitoring of station battery status. - - Support to the Operations Section and Maintenance Section by monitor-ing backscuttle activities, diesel generator surveillance and system restoration after power was restored. Interviews by the AIT indicated that, although the TSC was effective in carrying out its activities, the role of the TSC may not have been clearly understood by the Control Room personnel. The procedural aspects of administratively staffing the TSC are not clearly delineated. Addi-tionally, provisions are not made for the management of certain recovery actions from the TSC under circumstances where the TSC is not formally activated. The last major support activity involved the use of the Maintenance Section. Under the direction of the Maintenance Section Manager, this section implemented the actions necessary to troubleshoot equipment,
correct faults and return components and systems to an operable state. Major activities included washdown of switchyard insulators, restoration of the "B" emergency diesel generator, restoration of the Startup Trans-former, connection of a temporary high pressure air supply, and removal of main generator disconnect links to allow backscuttle of offsite power. The Maintenance Section Manager provided much of the coordination and ' direction of the recovery activity. This activity was somewhat delayed in getting started as the Maintenance Section Manager did not arrive on site until 7:30 a.m., many work force personnel were delayed in reaching the site by the adverse weather, and management and supervisory personnel needed to assess the recovery options. Starting at about 10:00 a.m., periodic meetings were held in the Maintenance Section Manager's Office to assess recovery progress and provide direction and coordination of site personnel activities. Attendees included the Maintenance Section Manager, Nuclear Operations Department Manager, Executive Assistant to the Senior l ! . -- - . _ _. I - _ ___. __ _ _ _ - _ [ s!, ! '
Vice president-Nuclear, Technical Section Manager, Assistant. Chief Main-tenance engineer and others. Interviews by the AIT indicate that no one
from the Operations Section or NRC attended these meetings. The BECo l recovery schedules prepared by the Maintenance Section Manager are presented in Appendix E. These represented the status and decisions from ! these periodic meetings. l The periodic meetings appeared to provide focus and direction to the LOOP recovery ef fort. The absence of the Operations Section may have impeded a a,) full understanding of recovery actions by operations personnel and meant that' planners may not have fully been able to consider operational aspects of their decisions. 5.4 Management Activities and Communications The AIT examined various aspects of overall management activities and-communications as they related to the LOOP event. These aspects included - overall control and coordination of activities, planning actions (before the event and during the recovery), compensatory measures taken in ,d response to changing conditions, control room activities and use of the l TSC. Also reviewed were management aspects of the backscuttle, the "B" '., emergency diesel generator shutdown and the loss of station air. ! Overall control of the plant in immediate response to the event and ! initial actions to notify ar.d recall support personnel were adequate, as ! described in Section 5.2. Interviews by the AIT indicated that the role ' and responsibilities of the NWE remai r.ed clear and consistent with j established practice and procedures. The NWE's clearly understood and l t properly observed requirements to assure reactor safety at all times ! during the event and subsequent power recovery activities.
Overall management of the recovery effort appears to be somewhat frag-mented over time. It was difficult for the AIT to determine who had , overall responsibility for managing the recovery. Early morning efforts ) by Maintenance Section personnel focussed primarily on the 345 kV switch-yard without clear management involvement or consideration of the back- ! scuttle path. In general, the Nuclear Operations Manager remained in the control room, except when attending meetings in the Maintenance Section ,' Manager's office. His operational management involvement was evident, ! however, his overall direction of station activities seemed less clear. I By late in the morning on November 12, management of the recovery effort appeared to reside primarily with the Maintenance Section Manager. Strong i direction and coordination from the Maintenance Section Manager was ,j evident throughout much of the remainder of the recovery, as exemplified l by the periodic meetings he chaired e.nd his issuance of BECo recovery documents. The overall recovery appears somewhat lengthy; however, this may be attributed, in part, to the weather, the cold shutdown condition of the plant (which did not mandate speedy action) and BECo management's stated directive to proceed slowly and deliberately during the recovery. \\ l - _. _ _. _ _ _ _ _ _ - _ _ _ - _ - _ _ _. _ _ _ _. _ _ _ - -. _ _ _ _ _. _. . - . . , ,, i , .8 s g . t Planning activities prior to the event were limited. It is not clear that BECo examined the overall impact of having significant c'omponents for a - large number of systems out of service, particularly in the light of the possible occurrence of analyzed events. For eka).ple, t.aking the shutdown transformer out of' service for an extended peribd end not removing the main generator disconnect links reduced the st'ation's flexibility in i vapanding ' k a LOOP event. Having several air compressors simultaneously out of' 'supice also reduced flexibility. ' ' Planning fo .he poss[oiiity of severe weather appeared limited and may be somewhat dependent or good practice or previous experience. BECo does have a high winds procedun for dealing with the potential for winds in excess of 75 miles pe) hour. Planning activities dur'ing Om reconry were slow sn getting start.ed; g owever the plannira actions by tbe Maintenance Section were a positive h dup,iin bringing focus to the recovery ef fort. Hourly meetings in the i Maintenance Section Marager's office were held to assure coordination. Planning for the demands of an extended event on personnel fatigue was generally acceptable. For example, the Maintenance Section Manager used a ' computerized data base to mcnipor ami adjust personnel work hours and the Operations Section maintained 'normsi control room watch rotations. qq y Several planning weaknesse-during the event were noted by the AIT. It was not clear that management, was moving aggressively to assure restora-tion of power capability it a timely manner. For example, the initiation of the backscuttle effort was not evident until mid-day on November 12. Another example, involves thet 'B" emergency diesel generator. When the cause of the failure of the current transformer circuit was determined, efforts could have moved more promptly to restore the emergency diesel generator. Another planning weakness contributed to poor communications between the operations and maintenance organizations. As described in Section 5.3, Operations personnel were not included in the Maintenance Section Manager's periodic planning meetings. Planning and control of insulator washdown was hampered by conflicting priorities. Initial washdown efforts were directed towards the path between the No. 355 'ine and the startup transformer (the shortest wash-down path); however, it is not clear that selection of this path con-equipment necessary{ li'.y of insulation resistance and high potential test sidered the availal to restore the startup transformer to operation and the necessary test c' ' checkout time requirements. Later washdown efforts were redirected towac s the path between the No. 355 line and the main transformer (the longer washdown path). Coincidentally, weather condi-tions improved and the washdown of the longer path was completed in a shorter time than the washdown of the path to the startup transformer. Planning activities during the recove ry included consideration of the status of station air and the actions necessary to obtain a tempora ry diesel driven air compressor and connect it to plant instrument air systems. Implementation of this installation was delayed by the onsite delivery of the air compressor; recovery of normal power supplies to station air compressors precluded the need for this installation. _ - _ _ _ _ _ _ _ _ -. Ml % . % ', '# vlp , . . . J .. - q ., - v ' ./ ' - N' , Lt
y < , ^ t,, , ' 'i Management of Na bachscuttle activity say hanpered by the presence of ,,' numrous tags 4 fectinj_.tbe an f li ary t ransformer. and main generator ' ' , ' iso phase buequick discenrg;t'lGks. Some of theaf tap were placed in support of the pending blaceut-dieself generator installation and some were left ever' tags from cenpletsj maintenance actions. p Strteter control of tags may, have. : reduced the ~ effqrt necessary tc cieu the tags at a ' { critical tima. I ' .,, Compensatory hueesures jn:pinentid by the 1:arsee dring the LOOP and ' recovery were satisfer. tory. Examples incluN , - Personnel monitur"ng'of emergency diesel ggnerators .fh.nitoring of sudp - , , Plytt tcurs and inspections - . Fire watcnes to compensate foc > fire suppression system realignments -
made as a result of the loss 'of station air to certain fire ' suppression system valves Activities in the control room and the TSC were gene-tity well managed by . ' licensee supervisory and management personnel. Interviews by the AIT d, indicated that control of personnel and communication 7s were generally '/ effective with two exceptions. The first concerne,i the number of L ! personnel in the control room. The continuity, pesence of management and L NRC personnel in the < NVP s c f:fice, in part, t ocessitated because of the backup telephow. comr:u r'pattarkthat were established with the ENS line initially incpdrable, 4 %rced the NWE cut of his office and may have B preventd g nth hom mmW ef fectively managing watch section activities. < Tli seconci control and con;munications weakness involved the shutdown of the "B" enegancy diesel generator (EOG). Although the COE and NWE were . both made ~ aware of'1 % "C" phase current transformer problem shortly I ' , before the EDG ws stsped 60 '11:35 a 4., they both lef t the control room / ar,a critical-time without clearly ascertaining the intentions of the NOS regarding the EDG. ThvCOE and NWE did not give sufficient attention to a vital power supoly an0 alternatively, both pursued tasks of lesser importance in /olvi ng the air compressors and turbine building closed cooling weep ' Coincidentally, the Nuchtar Operations Department Manager, who had been present in the control room much of the morning, also was absent. Consequently, the leadership in the control room was not suffi-dont to assure continuation of this v%al power source until a more mWdered decision regarding '!ts shutdowa could have been made. , c Licensee managemeat acted effectively and responsibly in administratively Oaffing the TSC. As discussed in Section 5.3, this contributed to ef fective management of tu fecovery. Interviews by the AIT indicate that the Mcensee believes the TEC could have been of greater value had it been staffed sooner. One a wect' of' the TSC staffing is noteworthy from a management standpoint. The hclear Operations Department Manager would normally be stationed in the TSC if the TSC had been activated pursuant to _ _ _ _ _ _ _ _ - _ _ _ - _ _ n; the Emergency Plan. In that the administrative staffing of the TSC is not prescribed by the plan, the Nuclear Operations Department Manager remained in the Control Room. Had he been in the TSC he may have been able to more effectively direct the overall recovery effort. At various times during the recovery effort the licensee made statements to NRC to the effect that " Power can be restored, if really needed, in a very short time." AIT interviews indicate that statements of this nature were generally without clear technical basis. Initially, power restora-tion was precluded by severe weather and lack of clear knowledge of conditions in the switchyard. Later, power restoration via the startup transformer would have been precluded by the need to properly test and checkout the transformer. Essentially, all knowledgeable personnel stated to the AIT that they would strongly recommend complete checkout of this transformer before attempting to place it in service. Power restoration via backscuttle was not immediately possible until disconnect links were removed, tags cleared, and insulator washdown was complete. 5.5 Operator Action The AIT reviewed actions of both licensed and non-licensed plant operators in response to the LOOP and during the subsequent recovery. No signifi-cant discrepancies were identified. Appropriate concern for reactor safety and procedure adherence was maintained. 5.6 Procedure Adequacy The AIT conducted a general review of portions of selected plant proce-dures that were of interest to this event. Interviews by the AIT revealed that several procedures are being reviewed by BECo to assure that they are . adequate. These procedures are: - 2.4.16, Distribution Alignment Electrical System Malfunctions - 2.4.144, Degraded Grid Voltage - 2.2.36, Instrument Air System - 2.2.37, Service Air System - 2.2.46, Control Room / Cable Spreading & Computer Room Heating, Ventilation and Air Condition system - 5.3.8, Loss of Instrument Air - 5.2.2, High Winds (Hurricane) 5.7 Radiological Aspects The AIT reviewed the radiological aspects of the LOOP and interviewed the Radiological Section Manager. The AIT also inspected certain radwaste areas in the Process Building affected by a minor water flooding event. __ _________ ___ _ _____ -- - -- _ _ _ . _ , Early in the recovery period the plant staff recognized that the LOOP prevented operation of various radwaste system components such as sump pumps. They also recognized that the loss of station air affected operation of many air operated valves in the radwaste system. Radio-logical Section, Operations and Technical Support Center personnel made a ' number of plant tours to monitor systems and components for leakage. - Particular attention was paid to sumps because leakage was accumulating { therein and could not be readily removed. At' 2:45 p.m. on November 12, plant personnel discovered minor flooding on the -1 foot elevation of the Process Building. This was caused by water backing up through the drain system and overflowing from the turbine building sump. At about the same time plant personnel discovered the ' ingress of ground water through seams and a conduit penetration in the West wall of the Process Building. This water was accumu'tating on the floor of the Monitor Tank Room. The principal source of this water was from a conduit penetration, which contained a conduit leading out to a small sump for the switchyard area (adjacent to the switchyard gate). The failure of the switchyard sump pump coupled with excess ground water in the switchyard (due to insulator washing and melting snow) resulted in this conduit flooding. In response to the minor flooding, which generally was no more than " puddling" on some of the floors in the radwaste area and in the radwaste area hallway, the plant staff placed a temporary sump pump in the' turbine building sump. This pump was powered by vital power and was used to transfer the excess water to the Chemical Waste Tank. Radwaste Section personnel successfully controlled the water ingress and sump overflow throughout the event. Site personnel estimated about 3000-4000 gallons were transferred to the Chemical Waste Tank. When power was restored, normal radwaste processing was re-established and the floor of the radwaste area hallway was decontaminated. (Initial contamination levels ranged from 1000-5000 dpm/100 cm ). Decontamination was completed within four hours. No airborne contamination was identified and no personnel contamination occurred during or after the LOOP event. No other significant radiological issues were identified by the AIT. Interviews by the AIT revealed that Pilgrim has had some previous diffi-culties with ground water ingress, both through seams in the Process Building wall and through the conduit penetration for the switchyard sump pump. This is caused, in part, by a high water table around the Pilgrim Station and by insufficient corrective action to deal with identified sources of leakage. This water ingress contributes to a radwaste proces-sing burden. Licensee personnel indicated that BECo plans to more , aggressively pursue courses of action to mitigate this problem. . _ _ _ _ _ - - _ - . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ -__-_ __- _ _____ __ _ _____ i k .; I 5.8 Conclusions and Recommendations The following. conclusions are made regarding human performance during the LOOP and subsequent recovery: - Operational response to the LOOP, event classification, notifica-tions, compensatory measures, radiological control activities, and operator actions to assure reactor safety were correct and in accordance with procedures and requirements. Overall management of the recovery effort was somewhat fragmented and - unclear. Communications between organizations and between individuals resulted in uncertain roles and some undirected actions. Administrative staffing of the TSC and formal recovery planning - activities were positive steps in providing focus and direction to the restoration of normal power. Procedural guidelines do not exist for the administrative staffing of the TSC. - Certain procedures governing degraded plant conditions (e.g., loss of power and loss of instrument air) may not be sufficient to clearly guide recovery actions from events of this nature. The following recommendations are made to strengthen human performance at the Pilgrim Station: - Strengthen the management guidelines and roles to assure a clearer and more positive direction to recovery activities following major events. - Strengthen communications practices to assure clear understanding and directed actions. - Re-evaluate emergency action levels regarding loss of onsite and offsite power for situations where fuel is loaded in the reactor vessel and RCf temperature is less than 212 F. < Develop and implement procedural guidelines for administratively - staffing the TSC to support the operating organization in situations where Emergency Plan activation is not appropriate. - Re-evaluate plant procedures that may be required in loss of power events to assure they are sufficient to guide recovery from degraded plant conditions. - Although not directly related to human performance, BECo should j continue actions to reduce ingress of ground water to the Process j Building and consequential radiological burden. The practice of having the main generator bus quick disconnect links - installed during an outage should be reviewed relative to the recent j event. j - - - - -- J ! - The plant configuration and equipment out of service for outage maintena'nce created operational inflexibilities during the event which, under other circumstances, could have had serious impacts on the operator's ability to cope with the situation. BEco should provide a review process for plant configuration and equipment in a maintenance status to assess plant and operator needs during outages to cope with outage operation oriented transients (i.e. loss of offsite power, loss of shutdown cooling). 6.0 Safety Assessment The safety implications of this event have been assessed for the actual ] occurrence as well as for the event under different reactor conditions (decay heat level) and as a station blackout precursor. 6.1 Reactor Safety Significance of Actual Event At the time of the LOOP, Pilgrim Nuclear Power Station (PNPS) had been shutdown for an extended outage for some 19 months. The plant was in a cold shutdown condition with the reactor coolant system temperature less than 212 F and reactor coolant pressure maintained at about 2 psig with the reactor vessel vented. Reactor core decay heat level was minimal due to the extended shutdown time and with the core being comprised of ore third new fuel. The decay heat was insufficient to heat the coolaat to a temperature necessary to perform pre-startup hydrostatic testing. The RHR system was being operated in a recirculation mode wher'ein heat was being added by RHR pumps to the coolant to maintain temperature at about 180 F. Reactor vessel water level was up at the 212 in. level well above the normal operating range. All four RHR pumps were available for service, only."A" RHR pump was running. Only one offsite power circuit through the stcrtup transformer was immediately available and all plant safety systems were powered from that source. Both emergency diesel generators were available in standby. A discussion of key system and equipment status prior to the event is provided in Sections 3 and 4. The loss of offsite power resulted in both a reactor protection system (RPS) actuation and primary containment isolation signal (PCIS). The latter caused isolation of the RHR which was restored after 39 minutes. During the next 24 hours, RHR was lost (isolated) two more times, once for 23 minutes and again for hours 27 minutes. The longest interruption of RHR occurred when the B diesel generator was taken out of service. RHR wasn't restored until after offsite power was restored. Because decay heat levels were so low, reactor coolant temperatures did l not increase during the course of the event when RHR was not available. l In fact some recorded temperatures decreased. Average temperature ] l decreased about 2 F. Heat losses through the vessel contributed to this ' reactor cooldown. In general reactor coolant and vessel metal , I temperatures remained around 170 F-180 F. Thus, at no time during the event, which lasted more than 24 hours, did the conditions within the reactor coolant system degrade to a potentially safety significant condition (i.e., coolant temperature was always less than 212 F). l (. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _________i _ _ - _ - _ ___ _ _ When the B diesel generator was taken out of service the operability of two other systems was affected. These were the instrument air and normal control room heating, ventilation, and air conditioning (CRHVAC). Air operated instrumentation was lost when MCC B14 which powers compressor K104B was de-energized. The loss of these non-safety systems should not be particularly troublesome since no credit is taken in the plants final safety analysis (FSAR).for the availability of either system during an accident. The loss of air pressure causes operability problems with some systems used during normal plant shutdown and in particular CRHVAC requires instrument air for proper chiller and damper operation. i However, the emergency air filtration system was operable, and occasionally used, during the event. It is not known if operation of CRHVAC was affected when air pressure was available but low during the event. Control room heat loads were such that somewhat uncomfortably high temperatures were felt by operations and NRC personnel present raising a concern about control room habitability and control room equipment operating temperature specifications (see AE0D C604). Because at least one diesel generator was operable at all times, control room heat loads during this event may have exceeded those expected during events involving loss of all AC power which would result in the loss of both safety and non-safety control room ventilation systems. It is believed that the control room temperature did not rise much above 90 F. The maximum operating temperature specification for equipment located in the control room is 120 F. The availability of the emergency air filtration system and the cool outside temperatures combined to help keep bulk control room temperatures well below equipment operating temperature specifications. FSAR analyses predict a maximum control room temperature of 114 F. Computer equipment cabinet temperatures were recorded on the process computer alarm printout which shows local temperatures of about 95 F at 3:04 a.m. and up to 108 F at 9:41 a.m. (about 7 hours after LOOP). By 4:45 p.m. cabinet temperatures were close to or below the alarm set point of 95 F. It is not known if these or other electrical equipment cabinet temperatures exceeded the highest recorded values. It is also not known how indicative these recorded temperatures are for other cabinets contain-ing essential safety equipment. No equipment failuras were reported which could be attributed to high temperatures. Therefore, the actual impact of any temporary or sustained loss of CRHVAC which may have occurred was nil. 6.2 Reactor Safety Implications at Hiaher Decay Heat Levels In light of past history at PNPS it is possible that the LOOP event of November 12, 1987 could have occurred either while the reactor was at power, or if shutdown, with substantially higher core decay heat levels than existed at the time of the event. If at power, a reactor trip would have occurred with the feedwater system and condenser lost for decay heat removal. High Pressure Coolant Injection (HPCI) and Reactor Core _ - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ - _ _. _- _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ I Isolation Cooling - (RCIC) systems would have been used for core cooling with steam being relieved to the suppression pool. Under these circum-stances the plant could remain on RCIC with Suppression Pool Cooling (SPC) for a considerable period of time. RHR could also be initiated when the reactor pressure and temperature had been reduced. Decay heat removal should not be significantly affected with a long loss of offsite power and no other system failures. During the LOOP event of November 12, 1987 at least two other noteworthy malfunctions occurred. The first being degradation and then loss of plant air systems. This malfunction could have affected CRHVAC and caused other BOP systems to be inoperable. Other than being a nuisance it does not appear to have affected any safety systems capability adversely. The second malfunction involved the B diesel generator current transformer (CT) circuit and its subsequent removal. from service. The removal (or failure) of either diesel generator results in the de energization of certain vital safety buses which results in PCIS, in particular a group III isolation of the RHR suction lines. Thus, just as RHR was lost several times for varying durations on November 12, 1987, it could also have happened if the reactor had been shutdown more recently with higher decay heat levels given a LOOP and one diesel generator failure. The reactor coolant heat up characteristics would have been different from November 12, 1987 if the reactor had recently been shutdown. While no reliable calculations of reactor coolant heat up rate for PNPS under these conditions are available, it is possible to make reasonable estimates. Past operating experience regarding loss of RHR at other BWRs has shown a reactor coolant heatup rate ranging from 2 F per hour to 22 F per hour under various conditions of reactor decay heat rate and initial reactor coolant temperature. The licensee has estimated a maximum heat up of about 60 F per hour for a recently shutdown plant in cold shutdown conditions. A loss of shutdown cooling procedure (2.4.25) was referred to early during the event of November 12, 1987. The procedure directs the operator to determine the cause and "take steps to provide a heat sink for decay heat". With regard to vital power the operator is alerted to check 4.16 kV Engineered Safety Feature (ESF) buses and 120v RPS-supplied power to PCIS logic. Additional instruction on loss of one 4.16 kV ESF bus is not provided. However, procedures 5.3.18 and 5.3.19 provide some applicable guidance for loss of safeguard buses Y3 and Y4, respectively. In parti-cular they state that a temporary modification to the high pressure iso- , ! lation logic for shutdown cooling would be required. During the event of November 12, 1987, technicians identified the appropriate circuits to be l jumpered and a temporary procedure was written to initiate that activity l had it been necessary to establish shutdown cooling with only one EDG operating. While it is not known how soon shutdown cooling would have been restored if the reactor decay heat rate was high, it is apparent that RHR could have been re-established in a time much shorter than the 14 hours it took to repower bus A6 had it been necessary. l ! _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ ____ _ _ _ _ - l 6.3 Station Blackout Safety Implications While the station blackout rulemaking has not been completed at this ! time, losses of offsite power and preparedness to cope with such eventualities including loss of all ac power at PNPS have been considered i in this inspection. The purpose was to assess the PNPS capabilities regarding station blackout, both in place and planned, and to determine any generic implications for the ongoing station blackout rulemaking activity. In summary, the proposed station blackout regulatory requirements include: (1) capability of coping with a station blackout of duration dependent on offsite power and emergency ac power reliability; (2) diesel generator monitoring, maintenance, and modifications as necessary to meet certain reliability targets; and (3) procedures which cover actions necessary to assure reactor core cooling during extended blackouts and procedures to effectuate timely restoration of ac power. The PNPS has experience 19 events involving loss of power from 345 kV lines in 16 years prior to the event of November 12, 1987. There have also been instances where power from kV lines was lost simultaneously with 345 kV sources during that period. The three total losses of offsite power had outage durations of 2 hr/40 min, 8 hr/54 min,* and 3 min. The gives an occurrence frequency for LOOP at PNPS of 0.187 if the November 12, 1987 event is not counted as a total loss of . ! offsite power and 0.25 if it is. The national average for nuclear power plant sites is about 0.08. Because the PNPS losses are primarily due to wind, snow, and possibly salt spray from the ocean, the losses at PNPS are typically longer than the national average of about 30 minutes. Technical findings developed through NRC analysis in support of the prepared station blackout rule show that longer duration losses of .' offsite power will most likely result from adverse weather and plants located in areas where severe weather can cause a LOOP with relatively high frequency may have a high station blackout risk. Durinf, the period 1983-1986 the PNPS diesel generators have had an average unavailability of about 0.025. This value is near the industry average of about 0.02 and corresponds with the higher of two diesel generator reliability levels identified in the proposed station blackout regulatory requirements. BECo has been participating in a pilot safety system unavailability monitoring program which has included the emergency ac power system. The program includes such activities as tracking and trending both equipment and system reliability performance and comparison to targets, assessment of major unavailability contributors, and proposed corrective actions to improve unavailability performance to meet targets. The program is in the demonstration and assessment stage of implementation and only limited information was made available for inspection at this time.
- As reported in NSAC/III and NUREG/CR-3992 l
_ ____________________-_ a - _ _ _ 64 Another aspect of PNPS station blackout capability that was reviewed involves procedures for maintaining core cooling during extended blackouts and procedures for restoring ac power. Since the event of November 12, 1987 occurred when the reactor was in cold shutdown, the emergency operating procedures would have been of. limited value if a loss of all ac power had occurred. The emergency operating procedures (EOP's) do address steps to be' taken for a loss of all ac power when the reactor is at operating conditions (steaming). These involve the use of HPCI and RCIC to maintain core cooling and directing recovery of electrical power. In j - fact, procedure 2.4.16 was used both initially and later in a temporary j procedure during the event of November 12, 1987. Procedure 2.4.16 is principally directed toward breaker alignments for events involving loss of. offsite power, LOOP with A or B diesel generator failure to start and loss of all ac power (station blackout). s For the latter, it describes de power alignments, cautions the operators on de power capacity and the need to load strip, and briefly discusses the premise that core cooling can be maintained fcr several hours using systems dependent on reactor steam and dc power. In addition, it describes the actions and breaker alignments for restoring power via the startup transformer or "backscuttling" of the unit auxiliary transformer, and.provides several procedures relevant to diesel generator operations and malfunctions. There are numerous other procedures referenced which operators are directed to when conditions warrant. It is not apparent that past operating experience, especially lessons learned from previous snow and wind induced LOOP, have been factored into the offsite power restoration procedures i n 2. 3.16. In fact, temporary procedure 87-252 was written to deal with the conditions on November 12, 1987. An integrated strategy for coping with a station blackout is also not apparent. This observation is based on the fragmentation of proce-dures, potentially many, which could be called upon to successfully cope with an extended station blackout. In these procedures conservation and possible replenishment of dc power and cooling water supplies are only alluded to. Actions to limit adverse environmental conditions (regarding i habitability and equipment operability) are also minimal. Plant response and the coordination of actions in a timely manner to optimize plant coping were not apparent. This is not to say that the current plant , l capability and procedures are wholly inadequate, but they do warrant up- ' grading. Discussions with watch engineers indicate that based on their experience and plant knowledge, they are prepared to take prudent actions, without spending substantial time sifting through and interpreting the interrelated procedures which would cover the full spectrum of operator i actions during a station blackout. As a remedy at least in part, to the procedural shortcoming identified above, PNPS has embarked on a program to develop detailed, integrated, plant specific station blackout procedure. While a draft version of these procedures was not made available for review, portions of a draft study which provides an assessment of PNPS response and capability during an extended station blackout were obtained and reviewed. The scope of the report appears to address the technical issues of concern raised by the i !
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ station-blackout rulemaking and technical findings. The study covers decay heat removal systems, instrumentation and controls, room heatup, de power, and modifications to enhance station blackout capability to cope with a station blackout of eight hours or longer. One design modification activity of note which was nearing completion at the time of the November 12, 1987 LOOP, involves the installation of a third - diesel generator as part of the PNPS safety enhancement program (SEP). This non-safety' related diesel generator, rated at 2000 kw, will provide a non-safety source of onsite ac power to the 4.16 kV safety buses. The unit is fully self contained, not dependent on any permanent plant systems for emergency operations, has an independent fuel tank, and a cooling radiator. The unit is skid ' mounted and housed in a pre-engineered enclosure to protect it from the environment. It can be made available manually from the cen ral room within one hour of a loss of offsite power. Although a -detailed review was not performed by the AIT, the safety enhancement program (SEP) diesel generator appears to be - capable of meeting the alternate ac power criteria as currently drafted in the proposed station blackout regulatory requirements. It has the potential to reduce station blackout risks at PNPS substantially, the staff has reviewed this installation as part of its review of the PNPS SEP. The staff review is contained in an August 21, 1987 letter from S. Varga to R. Bird. 6.4 Conclusions and Recommendations Conclusions The loss of offsite power event and other malfunctions which occurred at PNPS on November 12, 1987 did not represent a significant risk at any time during the incident. This finding is primarily based on the very low reactor decay heat levels which existed at the time. Cautious power restoration actions were prudent under the circumstances in existence on l November 12, 1987. At higher decay heat levels, equipment malfunctions which occurred would not likely have resulted in a more serious event although personnel actions and procedural adequacy would have been more important. The need i to write Tempora ry Procedures to restore shutdown cooling could have i delayed re-establishing RHR after the "B" diesel generator was secured and ' PCIS was initiated due to loss of power on bus 14. Loss of offsite power combined with loss or unavailability of decay heat removal systems through modes other than loss of onsite ac power sources may also be important at PNPS. The long time taken to restore offsite power probably could have been shortened considerably but many hours would be required to establish a reliable source of offsite power. The impact of the weather conditions on the availability of the 23 kV power source, had it been operable, is unknown, but previous operating experience shows both 345 kV and 23 kV power supplies affected during winter storms. Had the 23 kV source been available and unaffected by the weather, the significance of the November 12, 1987 event would be further reduced. _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - _ _ - _ _ - . _ ' Relatise frequency and long duration losses of offsite power at PNPS due to severe weather conditions (snow / ice, wind, salt spray) are consistent with the technical findings which support the station blackout rulemaking. ! Plant capability and procedures in effect on November 12, 1987 appear to need improvement to satisfy the draft requirements of the proposed station blackout rule. Ongoing efforts in the safety enhancement program which involve assessment of plant capability and development of plant specific station blackout procedures appear to meet the intent, at least in part, of the proposed station blackout requirements. Installation of the SEP diesel generator has the potential to sub-stantially reduce station blackout risks. This wi l,1 be especially important if switchyard improvements prove ineffective in eliminating long duration total losses of of fsite power in the future. Recommendations The loss of shutdown cooling procedure should be revised to address loss of power to either safeguards panel Y3 or Y4 both with and without offsite power available. Procedures for restoring offsite power should be reviewed against past operating experience, especially events caused by severe weather, and revised to reflect lessons learned and anticipated problems which may need resolution to optimize power recovery time. l . _ _ _ _ _ - _ _ _ _ _ _ ., ! APPENDIX A ENTRANCE INTERVIEW ATTENDEES NOVEMBER 16, 1987 Boston Edison Company (BECol H. Balfour, Training Group Leader R. Bird, Senior Vice-President, Nuclear M. Brosee, Maintenance Section Manager G. Edgar, N&H/BECo P. Farron, Staff Assistant J. Flannagan, Consultant R. Grazio, Field Engineering Section Manager P. Hamilton, Compliance Group Leader J. Howard, Vice-President, Nuclear Engineering and Quality Assurance S. Hudson, Operations Section Manager J. Jens, Radiological Section Manager R. Ledgett, Executive Assistant to Senior Vice-President Nuclear A. Morisi, Assistant Director of Outage Management J. Pawlak, Power Systems Gr,up Leader .V. Peters, Consultant K. Roberts, Nuclear Operations Department Manager E. Robinson, Nuclear Information Manager J. Seery, Technical Section Manager R. Swanson, Nuclear Engineering Manager B. Tucker, Senior Engineer E. Ziemianski, Training Section Manager ! U.S. Nuclear Regulatory Commission (USNRC) T. Kim, Resident Inspector Pilgrim J. Lyash, Resident Inspector Pilgrim C. Warren, Senior Resident Inspector Pilgrim _ ______ -__ _____ _- _ - APPENDIX B ' EXIT INTERVIEW ATTENDEES LIST NOVEMBER 20, 1987 Boston Edison Company (BECol R. Bird, Senior Vice President Nuclear W. Clancy, Systems Group Leader R. Deacy, Security Operations Group Leader R. Fairbanks, Design Section Manager P. Farron, Staff Assistant R. Grazio, Field Engineering Section Manager P. Hamilton, Cenpliance Group Leader S. Hudson, Operations Section Manager J. Jens, Radiological Section Manager R. Ledgett, Executive Assistant to Senior Vice-President Nuclear A. Lee, Onsite Emergency Preparedness Coordinator, EG&G B. Lunn, Senior Compliance Engineer P. Mastrangelo, Chief Operating Engineer J. Mattia, Quality Assurance Group Leader A. Morisi, Assistant Director of Outage Management K. Roberts, Nuclear Operations Department Manager E. Robinson, Nuclear Information Manager J. Seery, Technical Section Manager R. Sherry, Maintenance Group Chief Engineer A. Shiever, Operator Training C. Stevenson, Senior Compliance Engineer R. Swanson, Nuclear Engineering Manager B. Tucker, Senior Engineer U.S. Nuclear Regulatory Commission (USNRC) T. Kim, Resident Inspector Pilgrim J. Lyash, Resident Inspector Pilgrim C. Warren, Senior Resident Inspector Pilgrim i l -_____ -_ _ _ _ _ _ _ _ _ - _ - APPENDIX C PERSONS INTERVIEWED NOVEMBER 17-19, 1987 The below listed individuals were formally interviewed by the NRC team during the course of the inspection. Other personnel were contacted as the inspection interfaced with their area. R. Atkins, Senior Electrical Engineer R. Bird, Senior Vice-President Nuclear N. Brosee, Maintenance Section Manager J. Farrel, Electrical Lab Engineer J. Giar, Nuclear Watch Engineer B. Higgins, Electrical Lab Engineer J. Howard, Vice-President, Nuclear Engineering and Quality Assurance J. Jens, Radiological Section Manager G. Lafond, Staff Engineer, Maintenance R. Ledgett, Executive ~ Assistant to Senior Vice-President Nuclear B. Lyons, Nuclear Operations Supervisor M. Maguire, Senior Electrical Engineer P. Mastrangelo, Chief Operating Engineer J. McCann, Nuclear Watch Engineer S. Minahan, Nuclear Watch Engineer P. Moraites, Assistant Chief Maintenance Engineer K. O'Donnell, Staff Engineer, Maintenance W. Olsen, Nuclear Watch Engineer J. Pawlak, Power Systems Group Leader J. Purkis, Senior System Specialist K. Roberts, Nuclear Operations Department Manager J. Seery, Technical Section Manager S. Shidner, Electrical Supervisor J. Smith, Electrica Lab Engineer P. Smith, Technical Group Chief Engineer R. Swanson, Nuclear Engineering Manager K. Taylor, Nuclear Watch Engineer G. Van Epps, Nuclear Operations Supervisor D. Williams, Nuclear Watch Engineer _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ - _ _ - _ _ _ - . _ _ -__ - -_ ] j APPENDIX D INTEGRATED SEQUENCE OF EVENTS i The following sequence of events was assembled from various plant logs and personnel interviews, as noted. There are obvious variations in the times recorded in different sources for the same event. Where this was crucial
' to the understanding of the event, the most reliable source was used in the event analysis by the team. Where log entries were not clear or brief statements, they have been supplemented with commentary for clarity. SEQUENCE OF EVENTS PILGRIM NUCLEAR POWER STATION NOVEMBER 12, 1987 I LOSS OF OFFSITE POWER EVENTS
November 12, 1987 TIME EVENT SOURCE 1:27 a.m. 345 kV Line No. 342 Computer Alarm degraded voltage Log 1:27:05 a.m. Computer analog altrm received indicating 338.73 kV. Alarm setpoint is 341.50 kV. Alarm never clears indicating that voltage remains below reset value until the Loss of Offsite Power (LOOP). Operators not aware of alarm Operator and therefore did not enter Interviews Degraded Voltage Procedure (2.4.144) or monitor 345 kV line/ ring bus voltages to assess stability of 345 kV grid, 2:05 a.m. Loss of Offsite Power Relay operations for lines 342 and 345 kV line No. 342 355, 2:05:26 a.m. "A" to "C" to ground fault, ACB 103 and 104 open, ACB 104 was slow in opening, resulting in a transfer trip of breakers 105, 2130, and 312. I D-1 _ _ _ - _ _ _ -- . _ _ _ _ _ _ - - _. _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ - _ - _ _ _ _ -. _ _ _ _ -. __ _ - _ _ _ _ _ - _.. Appendix D TIME EVENT SOURCE 2:05 a.m. (con't) 345 kV line No. 355 "B" to "C" 2:05:34 a.m. phase fault ACB 1670 opens. Startup transformer differential leads'to ACB 102 opening. 345'kV line No. 355, breaker 1670 reclosed and reopened four more times, 2:06:56, 2:13:05, 2:15:01 and 2:15:55. REMVEC ordered the breaker manually opened. 2:06 a.m. 345 kV Yard NOS Log, ACBs 102, 103, 104 Operator and 105 open Interviews Full reactor scram Computer
due to loss of power Alarm Log. to RPS MG sets which 2:09:56 - are fed from 480 Vac 2:11:31 MCCs B22 and B23 Full PCIS due to NOS Log loss of power to 120 Vac buses Y3 and Y4 which are fed from 480 Vac MCCs B17 and B18 as well as de-energized RPS above. This isolates RHR SDC (Group III isolation) Full RBIS due to PCIS NOS Log EDG's "A" and "B" start NOS Log due to undervoltage on 4.16 kV Buses A5 and A6, EDG's tie on to Buses A5 and A6. Running air compressor K-111 stops due to loss of power to 480V, MCC B5 which results in depressurization of instrument air system. 2:12 a.m. NWE and electrical supervisor NOS LOG, respo.7d to switchyard relay house Elect. Supervisor to investigate event 0-2 - _ _ _ _ _ _ - _ _ _ - _- __ - l Appendix D TIME EVENT-SOURCE 2:20 a.m. Control room notified Nuclear NOS Log Operations Department Manager. REMVEC ordered opening of T-900 REMVEC Switching disconnect on line No. 342 in Order switchyard. 2:23 a.m. Re-energize RPS Bus "A" from MCC B-10 . When RPS Bus "A" is Computer re-energized, all "A" trips Alarm Log clear which allows reset of 2:22:55 a.m. - PCIS and restoration of RHR 2:23:44 a.m. ' in SDC mode. At this point neither "A" nor "B" scram has Operator been reset. Interview ' Reset PCIS and RHR Isolation Logged at 0221, but occurred after NOS Log
RPS Bus "A" re-energized above. ' This must be done before attempt- .i ing to re-open RHR valves 47 ' and 50 (RHR pump suction valves). 2:24 a.m. Started CRD Pump "A" NOS Log 2:27 a.m. Started RWCU Pump "B" NOS Log 2:30 a.m. Attempted Start of RHR Pump "A" NOS Log Received PCIS/RHR Isolation Operator Interviews Cause unknown 2:31 a.m. RPS MG Set "B" Reported to NOS Log be running on normal supply Operator i at reduced Voltage and Interviews Current NOS felt that "B" EDG might be overloaded and directed that 480 Vac Bus B4 be stripped and the feeder to B4 from 480 Vac bus B2 be opened. D-3 _ _ _ _ _ _ - _ _ - _ _ - , . - - _ --- -- - ---_---. _ _ _ _ _ _ _ _ _ . . Appendix D ! l TIME EVENT SOURCE 2:31 a.m. (con't) EDG "B" was loaded to 1300 KW. ! (Rated load 2,600 KW). ) "B" MG set was probably coasting down from the initial loss of power. ! 2:35 a.m. Shitted 120 VAC RPS NOS Log Bus "B" from "B" MG Operator set to 480 VAC MCC Interviews B10. De-energized 120 VAC RPS Bus "A". MCC B10 can only power one . RPS bus at a time. ! This caused a "B" scram signal Computer and allowed clearance of Alarm Log "A" scram trip contacts 2:36:57 a.m. - in RPS/PCIS. 2:38:55 a.m. Notified Maintenance Section Mgr. AD00M Log 2:36 a.m. Reset PCIS/RHR Isolation NOS Log Reopened RHR valves 47 and 50 Operator and re-established SDC. Interviews Moderator temperature slowly decreasing without pump heat from RHR. 2:45 a.m. Started RHR Pumps "A" and "C" NOS Log 0240, 0245 RHR Loop "A" flow analog Computer Alarm Log computer alarm clears 2:45:04 a.m. indicating 3267 GPM (one pump running). Subsequent pump start would not be indicated since flow already above alarm reset point. 2:46 a.m. Stopped CRD Pump "A" NOS Log 2:55 a.m. Notified NRC - Headquarters NOS Log Operations Officer (H00). ! D-4 { l i _ _ - - - _ - - - _ _ _ _ _ _ _ _ - - _ - _ _ - _ _ _ _ _ _ - ,__ - ______ _-_. Appendix 0 TIME EVENT SOURCE 3:00 a.m. NOM arrived on-site NOS Log 3:10 a.m. Air compressor K-104B placed NOS Log into service, essential System Specialist instrument air header repressurized Interview to 40-50 psig; insufficient pressure to reset. scram discharge header vent and drain valves. 3:15 a.m. Started RWCU Pump "A" NOS Log 4:00 a.m. Nuclear Operations Department Management Manager (NOM) notified Senior Interview V.P.- Nuclear and NRC Resident Inspector of event. 4:05 a.m. 99% of 345 kV switchyard Sr. Elect. Engineer insulators covered with snow Interview from top to bottom. 4:30 - 5:00 a.m. NOM notified Nuclear Information Management . Manager and Middleboro State Interview ! Police Barrecks (local Civil Defense) of event. NOM attempted to notify (Chief Operating Engineer), but phone inoperable. 5:00 a.m. Decision made to wash switch-Sr. Elect. Engineer yard insulators in preparation to Interview
' return startup transformer to service. ! 5:07 a.m. Stopped RHR Pumps "A" and Computer Alarm Log ' "C" and RWCU Pump "A" 5:07:05 a.m. In preparation for re-alignment of 4.16 kV NOS Log electrical system in accordance with Procedure 2.4.16 5:09 a.m. De-energized 120 VAC RPS Computer Alarm Log Bus "B" and energized 5:09:21, 5:09:27 a.m. 120 VAC RPS "A" from MCC B10 NOS Log 5:10 a.m. D-5 ) Appendix 0 TIME EVENT SOURCE 5:17 a.m. Energized 120 VAC RPS Bus "B" Computer Alar, Log from B22 5:17:14 a.m. NOS Log Fed from B2, B4, B22 through RPS MG set "B" 5:18 a.m. Reset Scram Computer Alarm Log 5:18:55 - 5:18:57 a.m. NOS log indicates reset 1/2 NOS Log scram, but computer log indicates 0515 full reset. Reset PCIS/RHR Isolation NOS Log 0516 , , Started RWCU Pump "A" NOS Log 0517 5:30 a.m. Started RHR Pumps "A" and "C" Computer Alarm Log-5:30:04 a.m.
Analog flow for RHR loop "A" NOS Log is 6880 GPM indicating two pumps running. 5:55 a.m. Blown fuses reported in NOS Log Analog Trip System (ATS) Panel Operator Interviews 6:00 a.m. Maintenance personnel ready to ' start washing insulators in Interview 345 kV switchyard to restore Sr. Elect. Engineer No. 355 line and startup transformer. 6:29 a.m. Blown Fuses in ATS Panel Computer Alarm Log replaced 6:29:27 a.m. MR 87-726 6:30 a.m. NOM notified Executive Management Assistant of event. Interview 6:57 a.m. NWE and REMVEC approve washing Procedure of insulators in 345 kV switchyard. 3.M.3-20 7:30 a.m. Mod Management Group to determine AD00M Log outstanding work which prevents using 23 kV shutdown transformers. 7:45 a.m. Stopped RHR Pump "C" NOS Log D-6 r__ _ _ _ _ _ _... [.' L Appendix D.- TIME EVENT SOURCE 8:00 a.m. "A" priority' maintenance Elect. Log Request to troubleshoot air Maint. Request compressor K-104-C for motor F&MR ground, reference MR-87-727. 8:05 a.m. Started RHR Pump "B" in NOS. Log Suppression Pool Cocling Mode (SPC)- 8:55 a.m. REMVEC ordered T-901 disconnect REMVEC switching. opened on line No. 355 order 9:02 a.m. 345 kV line No. 355 re-energized Sr. Elect. Engineer with switchyard disconnect Interview open, line tested good. 9:30 a.m. Senior Electrical Engineer AD00M Log ! and staff have checked startup transformer and found no problems; ' they are prepared'to wash insulators and re-energize from normal 345 kV source (estimated time 2 hrs.) i 9:40 a.m. REMVEC ordered disconnects 104A REMVEC Switching and B opened. Order 10:00 a.m. I&C Supervisor discovered "C" Operator phase ammeter on "B" EDG Interview Breaker ( A-609) reading zero. 10:45 a.m. Electrical Lab checking "B" diesel NOS Log generator, no indication on "C" phase. 11:00 a.m. Line No. 342 was made available to Sr. Elect. Engineer the Pilgrim station. Interview 11:20 a.m. Notified to secure "B" diesel due NOS Log to "C" phase problem. 11:23 a.m. Stopped RHR Pump "B" Computer Alarm Log SPC secured 11:23:04 a.m. Performed in preparation for NOS Log stopping "B" ECG. 11:20 a.m. D-7 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ - Appendix 0
i TIME-EVENT' SOURCE 11:25 a.m. RBCCW Cross-tied NOS Log Performed in preparation for i stopping "B" EDG Started unloading EDG "B" NOS Log 11:30 a.m. "B" diesel generator unloaded NOS Log and breaker tripped; reclosed automatically on the bus. NED wants testing of startup Sr. Elect. Engineer transformer, Electrical Engineering Interview ' was requested to initiate backfeed through the auxiliary transformer. Air compressor K-1048 stops due Operator and I to loss of power which results in System Specialist depressurization of essential Interviews instrument air header. '11:31 a.m. Reactor Scram Computer Alarm Log 11:31:36 a.m. Due to loss of A6 bus when EDG "B" secured. 11:35 a.m. "B" diesel generator stopped NOS Log using key lock switch, received full scram, lost shutdown cooling, TBCCW and instrument air. I 11:55 a.m. 125 and 250 voit de batteries are NOS Log ' placed on backup charger. I 12:35 p.m. Mode switch placed in shutdown NOS Log mode. 1:08 p.m. TSC partially staffed. TSC Log 1:20 p.m. REMVEC authorized opening of T-931, REMVEC Switching i T-930, and 1018 disconnects and Order i closing of T-900 and T-901 disconnects. 1:30 p.m. Doble crew ordered from Watertown Sr. Elect. Engineer to Plymouth to test startup Interview transformer. D-8 - _ _ _ _ _ - _ _ _ _ _ Appendix D TIME EVENT SOURCE 2:00 p.m. Parallel path activities in AD00M Log progress: (1) a diesel driven, oil free air compressor is expected to arrive onsite at 5:00 p.m.. The compressor will be connected through a previously approved modification; (2) Plan ! to Hi Pot the high side of the startup transformer, megger the low side and sample the oil for gases; (3) electricians are working toward "backscuttling" to feed through the main and auxiliary transformers. 2:15 p.m. Lines No. 355 and No. 342 were Sr. Elect. Engineer energized through Pilgrim Interview switchyard with startup transformer, and breakers 104 and 105 isolated. Power restored to lines No. 342 NOS Log and No. 355 by closing ACB's 102 and 103. Disconnect between startup RI Log transformer and bus opened. 2:20 p.m. 2:45 p.m. Health Physics reported R.P. Shift Log Radwaste elevation (-1 ft) flooded, water backing up through drains from turbine building sump. 2:47 p.m. Control room requesting fire TSC Log protection engineer to assess condition of plant deluge systems (loss of air), need to isolate and post fire watches 3:00 p.m. Tagging completed for backfeeding Sr. Elect. Engineer through auxiliary transformer. Interview Start isophase link removal. D-9 .. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - ____ . - - _ - _ - - _ _ Appendix 0 TIME EVENT-SOURCE ' 3:50 - 4:10 p.m. Fire truck moved to tie-in TSC Log point, hoses laid out, ready for backup. (Note: This was proposed, but never occurred.) 4i10 p.m. Main generator isophase bus AD00M Log links removed. 4:35 p.m. Notified problem with current AD00M Log transformer on "C" phase of "B" diesel generator was a burnt lead on the Watthour meter. Electricians will replace lug. 5:00 p.m. Isophase links removed; Doble Sr. Elect. Engineer crew arrived at station; "B" Interview diesel generator current trans-former lug repaired; meggering test performed on secondary of startup transformer and determined acceptable. Washing of insulators started to support backfeed through auxiliary transformer, i 6:00 p.m. Startup transformer gas test Sr. Elect. Engineer { results show transformer is Interview { acceptable; transformer oil sample taken for analysis. 6:15 p.m. Notified that red tags on AD00M Log MR 87-46-190 need to be removed to complete "back scuttling" on auxiliary transformer. I 6:33 p.m. Hi Pot tester being setup to TSC Log test startup transformer. j { 6:57 p.m. Doors between diesel generator TSC Log rooms closed; ventilation lineup was lowering "B" diesel generator { room temperature. 7:00 - 8:00 p.m. Doble crew onsite to test Elect. Log startup transformer. Electricians removing cables from bushings ) in preparation of test. I l D-10 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - - _ _ _ _ _ _ _. ._ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ - _ _ _ - _ _ _ _ - _ _ . _ - _ _ _ _ _ _ _ _ - _ _ _ . Appendix 0 TIME EVENT SOURCE 7:00 - 800 p.m. Secondary of startup transformer (con't) meggered, 100 megohms all three phases. 8:00 p.m. Insulator washing completed for Sr. Elect. Engineer backfeed through auxiliary Interview trans forme r. 8:30 p.m. Received permission to remove AD00M Log tags on MR 87-46-190. (Note: see 6:15 p.m. entry) The Maintenance Section Manager wanted the person responsible for the tags to personally notify the watch engineer that tags can be removed. 9:30 p.ra. Watch engineer received permission AD00M Log to lift red tags on MR 87-46-190 to allow "backscuttling." 10:00 p.m. Oil sample on startup transformer Sr. Elect. Engineer tested good per telephone Interview conversation. 10:50 p.m. "Backscuttle" failed because AD00M Log disconnect pin was stuck. Maintenance to disconnect-solenoid for pin. 10:58 p.m. Disconnect linkage freed, TSC Log disconnect closed. 11:09 p.m. Closed ACB 105 energizing the NOS Log auxiliary transformer. 11:15 p.m. Energized Buses A6 and B1 through NOS Log auxiliary transformer backfeed. Energized A-4 bus AD00M LOG 11:25 p.m. Energized A-3 bus AD00M LOG 11:45 p.m. Air compressor K-111 energized NOS Log 11:49 p.m. Pressurizing the instrument NOS Log air system. D-11 _ __ __ _ _ - ___ _ __-_ _ _ _ ____ _ _ _ _____ - _ - ' Appendix D l . TIME EVENT SOURCE 11:55'p.m. Energized B-4 bus. AD00M LOG . November 13, 1987 l 12:05 a.m. Air system attains normal Operator.and 102-108 Psig. System Specialist Interview 1:00 a.m. Reset Scram NOS Log, 1:10 a.m. Started RWCU Pump "B" NOS Log 1:20 a.m. Startup transformer work AD00M Log completed. 1:50 a.m. Started RHR Pump "A" NOS Log Restored RHR in SDC mode. 1:55 a.m. Identified problem with AD00M Log "B" EDG Lube Oil Prelube Pump Should have started automatically on restoration of power to MCC B18. 2:00 a.m. Reset RBIS NOS Log Testing completed on startup Sr. Elect. Engineer transformer. Interview, 3:15 a.m. Took reciosing of ACB 102 and NOS Log ACB 103 per REMVEC instructions REMVEC Switching opened'ACB 103 and 104. Order 3:30 a.m. Closure of T-931 disconnect REMVEC Switching authorized. Order 3:35 a.m. Closed in ACB 102 and 103 and put NOS Log reclosing to "on". Energized startup transformer. I 4:30 a.m. Preaction Fire System secured NOS Log to bring in temporary heater to warm "B" EDG D-12 h
F ~ ,,,, > Iri Appendix D TIME EVENT SOURCE 5:30 a.m. Startup transformer energized Sr. lect. Engineer and ready for' service. Interview
5:40 a.m. ACB tagging status: Elect. Log ' ACB 104 A&B disconnects open and j tagged ACB stuck breaker - off, no tag. MR-87-731 meggering and Doble testing of the startup transformer completed. .) Transformer is available for service if needed. 6:00 a.m. EDG "B" Lube Oil Prelube Pump F&MR 87-637 troubleshoot complete Pump drew locked rotor current and tripped when started. 6:40 a.m. Report that the "B" diesel generator AD00M Log prelubrication pump will be changed , .i 12:23 p.m. EDG Lube 011 Prelube Pump MR 87-737 replaced, returned to service TSC Log 1:50 p.m. Diesel generator pre-lubrication TSC Log pump is installed and being tested 2:40 p.m. Pre-lubrication pump on the "B" AD00M Log diesel is installed but there is some concern regarding the direction of rotation. l 5:50 p.m. MR-87-737 is complete, diesel pre-AD00M Log lubrication pump running satis-factorily, oil was warming. j 8:03 p.m. Started "B" EDG for NOS Log surveillance Monthly manual start and load per procedure 8.9.1. 9:30 p.m. A priority "A" MR is being written AD00M Log for the "B" diesel generator for leaking injectors. D-13 - _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - -_ _ _ _ _ . _ - _ _ _ _ _ _ . _ -.-. Appendix D > TIME EVENT SOURCE 10:00 p.m. Stopped EDG "B" NOS Log TSC Log Due to injector leaks November 14, 1987 12:15 a.m. MR issued on "B" diesel generator, NOS Log tagging out "B" diesel. 1:35 a.m. Started to energize Al and A2. buses NOS Log via the startup transformer.. Buses A3 and A4 on the startup transformer. I 1:45 a.m. Started Comdensate Pump "B" AD00M Log 2:05 a.m. Started /SSW Pump "B" AD00M Log NOS Log 2:07 a.m. Bus A6 transferred to the startup NOS Log transformer. l 2:45 a.m. Stopped RHR Pump "A" and RWCU NOS Log l-Pump "B" TSC Log ' 2:55 a.m. 4.16 kV Bus A5 transferred to TSC Log S/U transformer AD00M Log 3:00 a.m. Started "B" RWCU Pump TSC Log NWE Log 3:30 a.m. "A" diesel generator off the line, NOS Log 3:35 a.m. Reset RBIS NOS Log j 3:45 a.m. Secured TSC support. TSC Log 4:35 a.m. Opened ACB 104 & 105, secured AD00M Log backfeed 4:45 a.m. RBCCW LOOPS split NOS Log 5:30 a.m. Closed ACB 104 AD00M Log 2:50 p.m. Started EDG "B" for NOS Log Surveillance
f-f .. D-14 't -. -_-___ -__ Appendix D q , 3:16 p.m. Stopped EDG "B" due to Problem NOS Log with Bolting on Fuel Rack. 3:51 p.m. Started EDG "B" for Surveillance NOS Log 4:15 p.m. Stopped "B" EDG NOS Log 5:00 p.m. Restored "B" EDG Preaction BECo Chronology Fire System. i 5:08 p.m. Started EDG "B" for Surveillance NOS Log
a 7:55 p.m. Stopped "B" EDG NOS Log 11:15 p.m. EDG "B" Declared Operable BECo Chronology l l l l l , l i D-15 l m-. _ my. _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ .- _ _.. _ _ _ _ _ _. _ _ _ _ _ _ _ _ __ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ _______ _ _ _
- k"MAPPENDIX'E-1
. BECO REC 0VERY SCHEDULE , _... g .. _. . - - _ . g... .... . k >o - - - e 2m-o. . - a. "Z e< N - 9 3' D . ed - . k - .. ,p 61J c3 E a F l: g ._ m C s= a. E / [ .k g.) > ,a . E Mk W g~ -% e b .; . S-
O - d - a .. .. j 1 1 -
h... g ... _. .. _
. . . _ o w n _. 5'... g ..... . . - . _. .. _. ..... - n -. .. .._ . . _.... _. _.... _.._ _2 -_ . -. _.. ... _ ...... _. _. .b . . _w g- .- , -. - .-
- ..
(. E . r r l- - e -- -- - --- %, - - - - - - - - - ~ - - ~ - - t a . , .. _.. _ .. .._ _.._..... - - ,--- . . _.. ..... _ _.. _ _. . _.f. '. _. _.... _ d. ___L% ... .. -. _ _. -....-. . ' ... ,g .. -. 4. _.. , _.. . .. =_ y.. . - _ a . -... -.2 .- .*-.. ... o -. . o r A c
. . ' I o . - .. ._. . ._. . _.. _...... _ - _ _. . - -.. _ -.. -... _ - . f, g-- _,, . Y - .,f d w a g. . _ _ . _ _ .x,______. ..
.}_ ._ __ _ _.. __ ___g.g. _gg _. - _._ $ p_g.__. __.,_ g_).__ -. -_. .-.] 3 *_ _... _. [ .v _ g"..._k
._ . c' ul __..;_. o - cd v1 _ ..... . ' . .. d .O M ..., - _ -..d _... - _.. _ ..d _3 l y f*1 e f ,.. _ '. _ c 2 . _ _., .____-..w ._ .. _ _. g _ _._. o.. _3 . ._ _. _ _.
.,I. .- ..f _ . .- - . - x . _ _. _., -. - - .. _.... .. 3. _ ~......
. ... s _.. . ._ ... _g .. _. , . c. -..- o._. +.n-p ..m ._. -..e. .x . ._. x .._..o. . .._ . 0 ... _ _ _
- _ _.
.. _. _. _ g:. _.. -. -.. _. . . -_ .. _ _.
,8 _ _._e. g _.:Y_g _. __g 3._ _. -...Q-._.____._.__., t '. E - .. -.g. ,.. _. .... _. _.. _.. - s - .u. , .._ . _ . _... _. _ _ _ _ _.. __ _I~-d _ -. _ _ _... _ . - .. -.. _ _ _ _ _ _. _ - -... - .. _ _ _ _ _ _. M _ _ _ _ R_q', . ' - > . _-_. . w._. ___ LA ' 4.__ _. s _ _ _ _ _ _ _ _ _ _ _ _ _. __ - , i _ -.. _ .-._m..---._..-.__--.~.- -. _.. J _ _ - _ _. _ " APPENDIX E-2 BECO RECOVERY SCHEDULE h\\N {- - ' , 7. e .oC '
- _
l.n >C 9.NJl4 B c H e @ zw> M_ r ca _ g e - I- 'ogy'il a l-.s si E - E $ > b b .f - / h a-
% s a cm = - C4 y N U
E I . g-A < - - .g % <t m Q k4 ~ gd <#9 e o ea~o , w R a g~ Ef Ef 'wna Q f-gs wE ~ s
w n s G N a g- -> p ga v n g5 $ ~ -
. / g_
- .,g?)
a _ .__0 e x i t-a.__.....___.._y" o, . g .- _-. c . n ,._ _ I g .- _._ _ - __ J n.. _.. _. .. . g_ g l._g'o q .q, _ _ _. __. C.__) c 2 a e o j_.. ' + %q,u@ j 45 =d d U ' re L,, 3 e am M re - U m S k T Q - a &_ p._ g. g . y -. _. ha ? g:; - e ... " a t-r "8 0', ' r (* mh2 " E y ~y m ' ap o-E r
u N,; Hs p { e-b2 l* 'in & 00s y
' F 3- <o-e B e er
- e o
- t S a S E,3 - p s a< M
- E g a
b; g ~ -- - __. _ _ _ _ - _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ APPEf1 DIX E-3 Cge - g9 ) BEC0 RECOVERY SCHEDULE $ d O i cg ! ! ! s F %a i 1.1 e b g 3&A, = g . >
b
. e c
.. y 1 F s 8-et a s - - - - - - -? 'g g l Ws $ R-a.- p p L g .. k a e Og l g_ c g i I s i gE E a, l l". E l-E i g p, lls d ! F l<> gm I -, " a,
- e e
rl%._-g-i-___ bgg-* i e i-a p_ -- - V l h l g ~~i " if. ap' B I i i T e o a a - u o I s - h-S,o
IJ
" hh h----E -$ ! E 62 .<> V $L$ V f ?a' i, , " bo g~' - . F # d HI ! it-L.4p t a s e < 8, m e A E s w c a = = '* ^ .- _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _. _ _ _ _ _ _ _ - _ _. I APPENDIX F INSTRUMENT AIR SYSTEM INTERRELATIONSHIPS High Pressure Air System A loss of high pressure air affects the following systems. Clean radwaste system - seal air is lost to the flat bed filters, F-5
Reactor water cleanup (RWCU) system - blowdown air for the clean-up
filter demineralized is lost. Condensate demineralized system - sluiting air is lost
Solid recovery system - air to the spent resin transfer pump and the
solid recycle recovery pump is lost Screen wash system - air to the intake structure screen wash bubbler
is lost Non-Essential Instrument Air A loss of non-essential instrument air affects the following systems: Resin cleaning system - air operated ( AO) valves in the resin
cleaning system will shut, except the connection between the cement feeders to the mixer / feeder which will fail "as-is"; air to the dust collector, which supplies air to the cement bin, is lost Clean radwaste system - clean radwaste flatbed filters and instrument
seal air is lost; level indication for the radwaste tanks is lost Radwaste collection system - turbine building equipment drain sump
pump isolation valves fail shut; treated water hold up tank instrumentation and valves are inoperable Primary cor.tainment - air is lost to the MSIV's, the MSIV
accumulators provide reserve air pressure to shut the valves; torus make-up valves fail shut; and, reactor building / torus vacuum breakers will fail open if accumulator pressure is lost. Turbine building closed cooling water - Valve A0-4160, cooling water
to mechanical vacuum pump seal water cooler, fails open. Condensate demineralized system - all valves should remain "as-is"
except for the sluicing valves which will " fail-shut" Demineralized water system - demineralized water (DW) transfer valve
from the makeup demineralized system to the DW storage tank will " fail-shut" as will the transfer valve between the DW storage tank and the condensate storage tank; city water supply to the filter valves " fail-shut" as will the discharge from the acid pumps; and, level indication will be lost for the caustic storage tank and the acid storage tank. F-1 _ _ _ _ - _ _ _ _ _ _ _ _ _ _ - _ _ _ _ - _ _ _ _ _ - Appendix F Diesel oil system - diesel oil day tank isolation valves fail open
Fuel pool cooling system - fuel pool make-up valve fails "as-is";
skimmer surge tank level indication is lost Essential Instrument Air When essential instrument air is lost the following major systems are affected. Compressed air system - the following valves fail in the closed
position: A0-4365 - non-essential instrument air isolation A0-4353 - HP/LP service air cross-cor.nect isolation A0-4356 - essential instrument air isolation to containment atmosphere control system A0-4310 - instrument air dryer / filter bypass A0-4350 - HP service air isolation Main condenser vacuum - the main condenser vacuum will decrease as
the steam jet air ejector steam supply valve closes; the main condenser vacuum breakers will fail as is Feedwater heating - bleeder trip valves fail shut and spill valves
fail open thus eliminating all feedwater heating; the level control
valves fail closed and the high level dump valves fail open, diverting all drainage to the main condenser Cooling water - reactor building and turbine building cooling water
heat exchanger bypass control valves fail closed, providing maximum cooling to the RBCCW and TBCCW cooled components; the RBCCW and TBCCW head tank make-up valves fail open Off gas system - the off gas system isolation valve to the stack
(AO-3751) will fail shut Control rod drive system - the flow control valves will close, scram
valves will open under spring pressure (inserting the control rods) and the instrument volume vent and drain valves will close Feed and condensate - Feed regulating valves fail "as-is"; the
condensate pump recirc flow valve fails open; the feed pump recirc valves fail open; and the main condenser vacuan breakers fail 'as is" Standby liquid control poison tank level indication is lost
F-2 - - _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ .. _ _ _ _ _ _ _ _ _ _. _ _ - _ _ _. __ -_ Appendix F. Turbine Building Closed Cooling Water The TBCCW system supplies cooling water to cool the compressed air. It also condenses moisture in the air to aid in moisture removal. Losing TBCCW may cause severe damage to the compressor. If the compressor is started without cooling water, it must be shutdown immediately and allowed to cool down prior to initiating cooling water flow. F-3 - _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _. _ _. ._, _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ - _ _. _ _-_ _ ____ __ _ __ _ _ _ _ _. APPENDIX G SEQUENCE OF EVENTS LOSS OF INSTRUMENT AIR Approximate Time Event Source November 12, 1987 2:06 a.m. Loss of offsite power occurs, which NOS Log causes running air compressor K-111 to stop. Nuclear operations supervisor (N0S) refers to procedure 5.3.8, Loss of Instrument Air. 2:20 a.m. Air system block valves actuate to Operator and - 2.:25 a.m. isolate essential instrument air header System Engineer from high pressure service and non-Interviews essential instrument air headers. 2:35 a.m. Essential instrument air header Operator and - 2:45 a.m. pressure continues to drop off and System Engineer reaches essentially zero psig. Interviews 3:10 a.m. Compressor K-104B placed in service; NOS Log essential instrument air header commences System slow repressurization to 40-50 psig; Specialist insufficient pressure to reset scram Interview discharge header vent and drain valves, rated to function at 70-110 psig. 6:00 a.m. Systems Group NSSS mechanical lead System senior system specialist arrives Specialist onsite. Interview 6:30 a.m. Control room operator experiencing AD00M Log problems with station air system because only one compressor available powered off essential power bus. 7:30 a.m. Nuclear operations department manager AD00M Log (NOM) directed systems group to investigate availability of clean air compressor for temporary tie-in and to pursue options for restoring air system pressure. G-1 l .- _ _ _ _ _ _ _ _ _ _ _ _ _ - _ - _ _ _ _ _ _ _ _ _ Appendix G 8:00 a.m. Priority "A" maintenance request written System l to repair compressor K-104C; determined Specialist no suitable (oil-free) portable air Interview compressors onsite. '8:15 a.m. Vendor (Atlas-Copco) contacted to locate System compressor Specialist Interview 9:00 a.m. Atlas-Copco unit located in Connecticut System Specialist Interview 9:30 a.m. Senior system specialist for air systems System-10:00 a.m. arrives'onsite; conducts verification Specialist walkdown of air system response functions Interview and status, investigated alternate means of pressurizing header, recommended securing compressor K-104B to depressurize air header. 10:00 a.m. Atlas-Copco driver and transportation System arranged. Specialist Interview 11:30 a.m. "B" emergency diesel generator is NOS Log tripped manually which causes Operator compressor K-104B to stop. Interview 11:50 a.m. Essential instrument air header Operator and - 12:00 a.m. pressure drops off to essentially System zero psig. Specialist Interview 12:00 p.m. Portable compresser unit in transit, System ETA 5:00 p.m. Specialist Interview 1:40 p.m. Air compressor K-104C is tripping TSC Log on low oil pressure during post-maintenance testing of breaker terminal repairs. 1:43 p.m. TSC assesses compressor K-104C oil TSC Log
- 1:51 p.m. pressure problem; pressure should be ! 35 psig but is not building up beyond 4 psig. G-2 _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ _ _ _ _ _ _. _ - _ _ _ - _ _ - _ _ _ - Appendix G 2:23 p.m. TSC engineers dispatched to identify TSC Log ' outside/onsite available air sources; reported a compressor outside the radwaste truck lock; researching its capability (capacity, oil free?) ' 3:00 p.m. Air compressor K-104C tagged out of NOS Log service; was tripping on apparent low oil pressure. 3:24 p.m. Control room briefed on fire TSC Log protection air-operated valves and fail positions and accumulators. 6:22 p.m. Portable compressor arrived at the TSC Log site gate. 6:30 p.m. Compressor K-104C. work continuing Mechanical to replace the oil pump. Installing Maintenance Log temporary modification tie-in for Atlas-Copco compressor. 7:07 p.m. Installing temporary modification NOS Log 87-30, Temporary Instrument and Service Air Supply. 7:10 p.m. Portable unit onsite. 9:22 p.m. Temporary modification (TM) 87-30 and TSC Log
- 9:35 p.m. associated safety evaluation for installation /use of portable unit is approved by onsite review committee and NOM. 10:30 p.m. Operations, section completed TM 87-30 Operator and valve lineup, systems group received System direction from control room to keep Specialist portable compressor in standby mode. Interview. 11:09 p.m. Offsite supply breaker ACB-105 NOS Log is closed. 11:45 p.m. Air compressor K-111 re-energized. NOS Log 11:49 p.m. Pressurizing instrument air system. NOS Log G-3 L____--____ _ _ _ ___ _ .-. _ _ _ _ _ _ _ _ _. _ -. _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ________ Appendix G-November 13, 1987 12:05 a.m. Air systems attain normal 102-108 Operator and psig operating pressure band. System Specialist . Interview 1:30 a.m. Compressor K-104C tested satisfactorily Maintenance 'following repair of electical problem Request and oil leak. 9:30'a.m. Compressor K-104C tags /isolations Maintenance removed and returned to. normal lineup. Request 10:40 a.m. Compressor K-104C repair is completed. TSC Log I i l l G-4 _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _ - _ _ - w APPENDIX H ACRONYMS AND INITIALISMS > ACB Air Circuit Breaker AD00M Assistant to Director of Outage Management AIT Augmented Inspection Team ANSI American. National Standards Institute ATS Analog Trip System CR0 Control Rod Drive EAL Emergency Action Level EDG Emergency Diesel Generator F Degrees, Fahrenheit , GPM Gallons per Minute ! kV Kilovolts LOOP Loss of Offsite Power MCC-Motor Control Center M3 Motor' Generator NED Nuclear Engineering Department NOD Nuclear Operation Department NOS Nuclear Operations Supervisor NWE Nuclear Watch Engineer PCIS Primary Containment Isolation Signal PSIG Pounds Per Square Inch, Gauge RBCCW Reactor Building Closed Cooling Water RBIS Reactor Building Isolation Signal RCS Reactor Coolant System REMVEC Rhode Island, Eastern Massachusetts, Vermont Energy Control RHR Residual Heat Removal RPS Reactor Protection System RWCU Reactor Water Cleanup SDC Shutdown Cooling SPC Suppression Pool Cooling STA Shift Technical Advisor TBCCW Turbine Building Closed Cooling Water TSC Technical Support Center Vac Volts, Alternating Current Vdc Volts, Direct Current l _ _ _ _ _ - _ - _ _ _ _ }}