Text

Special Augmented Insp Team Rept 50-354/86-50 on 860925-1003.Major Areas Inspected:Root Causes for Problems Identified by Licensee During Loss of Offsite Power Tests on 860911 & 19
	ML20215N781
Person / Time
Site:	Hope Creek
Issue date:	10/28/1986
From:	Wenzinger E; NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I)
To:	;
Shared Package
ML20215N776	List: IR 05000354/1986050; ML20215N775; ML20215N787;
References
	50-354-86-50, NUDOCS 8611070267
	Download: ML20215N781 (54)
	v • d • e

O

,

v U.S. NUCLEAR REGULATORY COMMISSION

REGION I

Report N /86-50 Docket N Licensee Public Service Electric and Gas Company Post Office Box 236 Hancocks Bridge,.New Jersey 08038 Facility: Hope Creek Unit 1 Inspection At: Hancocks Bridge, New Jersey Conducted: September 25 - October 3, 1986 Inspectors: J. Shediosky, SRI, Millstone 3 H. Ei.chenholz, SRI, Yankee J. Wiggins, Chief, Materials and Processes Section P. Eapen, Chief, Quality Assurance Section T. Koshy, Reactor Engineer L. Stanley, IE Consultant C. S ulten NR Approved By:

"E'. C. Wenifnger,

~

h f g rojects Branch No. 3 A3[/h

' Dafe (Team. Leader) V Summary: Augmented Inspection Team Inspection Conducted On September 25 - October 3,1986 (Report No. 50-354/86-50)

Areas Inspected: Special team inspection to determine and evaluate the root causes for the problems identified by the licensee in the course of conducting Loss of Offsite Power tests on September 11 and 19, 198 Results: One noteworthy concern was identified regarding the adequacy of Bailey logic testing and operation. Several minor design and testing problems were also foun PDR ADOCK 05000354 G PDR

. - . .. _ _ _ _ _ _ _ _ . _ -

o

TABLE OF CONTENTS PAGE Introduction.................................................... 1 Augmented Inspection Team Formation........................ 1 Charter for Augmented Inspection Team...................... 1 Augmented Inspection Team Inspection Plan.................. 2 II. Summary and Conclusions......................................... 2 Summary.................................................... 2 Conclusions................................................ 4 III. Assessment of Results for the Loss of Offsite Power Tests....... 4

, Equipment Failures......................................... 5 Preoperational Test Deficiencies .......................... 8 Indeterminate and Unresolved Observations.................. 11 Observation Error and Other Non-Problems................... 13 Design and Design Control Issues........................... 16 Procedure Adequacy Issues.................................. 21 Construction and Manufacturing Issues. . . . . . . . . . . . . . . . . . . . . . 23 Operator Error............................................. 25 Security................................................... 25 Training................................................... 25 IV. Other Investigative Efforts..................................... 27 Other Related Events at Hope Creek Generating Station...... 27 Recent Events at Other Plants.............................. 33 Hope Creek Generating Station Preoperational and Startup Test Program............................................. 37 FSAR Commitment Tracking................................... 40 Regulatory Guide 1.97 Conformance.......................... 41

'

Redundant Reactivity Control System........................ 42 Overview by Quality Assurance During Startup Tests......... 43 B a i l ey Lo g i c Mo d u l e s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Bailey Logic Module Failure Rate........................... 45 Interdependency Between Automatic and Manual Protective Functions................................................ 48 In-Situ Testing Capability................................. 48 Field Programmable Logic Array Programming Errors.......... 49 Staple Jumper Placement Errors............................. 49 Bench Test Capability...................................... 49 Radio Frequency and Electromagnetic Interference Capability 50 STAG Field Programmable Logic Array Programmer Calibratio Failure Analysis........................................... 50 VI. Overview of the Cold Loss of Offsite Power Test on October 2, 1986.......................................................... 50 VII. Exit Meeting.......................................... ......... 51 i

. .- . __ .

--

. .- _ - _ ._ . . . .-. _ - . ._ . . .. _ . _

a

,

s

.

ATTACHMENTS Attachment 1 -. Memorandum to W. F. Kane and S. D. Ebneter from Thomas E. Murley,

, " Augmented Team Inspection -Design and_ Test Control Problems at Hope Creek Generating Station," Dated September 24, 198 Attachment 2 Loss of Offsite Power Test Anomalies

.i I

,

f I D

l l

a

!

l t

a

4

' ii

_,., , r.-- , , - , - , - - ,.,,, _ , , -- ,-y,

-

y-,

a e

I. Introduction Augmented Inspection Team Formation As part of its power ascension test program, the Hope Creek Nuclear Generating Station conducted a loss of offsite power test on September 11, 1986 from approximately 21.5% of rated power. Twenty-four (24) in-dividual test . observations ~were identified by Public Service Electric and Gas Company from this test. A second loss of offsite power test at Hope Creek was conducted on September 19th from a reactor shutdown con-dition (T <200 F), and seventeen (17) additional test observations were identified. On September 24th, an Augmented Inspection Team ~ consisting of the following members was formed to investigate and evaluate the test Aservations:

Team Leader: E. C. Wenzinger, Branch Chief, DRP, Region I Team Members: J. Wiggins, Section Chief, DRS, Region I P. Eapen, Section Chief, DRS, Region I H. Eichenholz, DRP, Senior Resident Inspector, Yankee T. Koshy, DRS, Region'I C. Schulten, NRR Representative J. Shedlosky, DRP, Senior Resident I'spector, n Millstone Unit 3 L. Stanley, IE Representative (Consultant)

The Augmented Inspection Team arrived onsite on 5eptember 25, 1986 and began evaluating the available data. An exit' meeting was held onsite on October 3, 198 e

' Charter for Augmented Inspection Team , .-.

-

,

A charter for the Augmented Inspection' Team was formulated and trans-mitted from T. E. Murley to W. F. Kane and S. D. Ebnetet on September 24, 1986 (Attachment 1). The inspection objdetives and scope are as follows: /- Conduct a timely, thorough, and s'ystematic inspection related to the test anomalies that occurred at Hope Creek Generating Station during the loss of offsite power testin'g on September 11 and 19,-

198 ,.. . .g

~ Assess the safety significance of the events and c'ommunicate to Re-gional and Headquarters management the facts and safety: concerns related to the test problems identifie . Collect, analyze, and document all relevant data and factual infor-mation sufficient to determine the causes, conditions, and circum-stances pertaining to the event #

-._,._..r _., , . _ _ . , _ , , _ , - - - - .

_

4 2 Develop a list of the test anomalies and assess and evaluate the significance.of these identified anomalie . Assess and evaluate why the preoperational test program and subse-quent surveillance testing did not identify these problems prior to the loss of offsite power test . Determine if, and to what extent, inadequate design controls, QA oversight, and training may have contributed to test anomalie . -Assess the extent to which the root causes of the test 'nomalies a

may impact other plant systems not addressed in'the loss of offsite power testin Augmented Inspection Team Inspection Plan The Augmented Inspection Team inspection plan had three primary objec-tive . ~ AccoEplish the established charter by reviewing and evaluating each observation from the hot loss of offsite power test on September 11, 1986 and the cold loss of offsite power test on September 19,

'198 . Examine the individual test observations for possible effects that involve the Bailey solid state logic module as an extension of pre-vious NRC inspection . Examine the Hope Creek design, procedures, test records, and other documentation related to issues that have been previously encoun-tered at Hope Creek and other plants, such as accident monitoring instrumentation, vacuum breakers, circuit breaker coordination, control of instrument root valves, and testing of standby liquid control squib valves to identify weaknesses common among these issues and the loss of offsite power test observation The results of this inspection are described in the following sections of this repor II. Summary and Conclusions Summary On September 11, 1986, Public Service Electric and Gas Company performed a hot loss of offsite power test (TE-SU.ZZ-311(Q)) at the Hope Creek

-

Generating Station from approximately 21.5% power. The loss of offsite power test is an important part of the power ascension test program re-conmended by Regulatory Guide 1.68, " Initial Test Programs for Water-

o N

0 :

,

) 3 s \

Cooled Nucle'ar Power P.lants." Its purpose is to demonstrate whether the plant response ,is satisfactory and in accordance with the plant design for concurrent 16s1 of the turbine generator and all offsite power source The Hope Creek Generating Station hot loss of offsite power test was in-

'

.

,, itiated with the turbine generator loaded to 165 MWe. The first indica-tion of an unsatisfactory plant response was the failure of emergency diesel generator C'.s output breaker to close automatically. Soon after, an observed failure of the reactor auxiliary cooling system coincident with increasing drywell pressure resulted in the loss of offsite power test being aborted by the licensee. Normal offsite power was then manu-ally restored to the statio Twenty-four observations were made by Public Service Electric and Gas during this tes These observations occurred during the time from initiation of the hot loss of offsite power test until the reactor vessel water level and pressure were controlled and the reactor scram was rese 'The most significant observations on September 11, 1986 were: (1) emer-gency diesel generator "C" output breaker failed to close; (2) MSRV posi-tion indication was lost; (3) power supplies for neutron monitoring source and intermediate range monitors' detector drives and main steam line acoustic monitors were lost; (4) 17 control rods did not provide a normal full-in position indication; (5) reactor auxiliary cooling sys-tem flow was lost; (6) emergency diesel generators "A" and "B" governors transferred isochronous (frequency control) to speed droop (load control)

mode without operator action; and (7) the "B" safety auxiliary cooling system pump failed to auto-star On September 19, 1986, Public Service Electric and Gas performed a cold s loss of offsite power test (TE-SU.ZZ-313(Q)) at Hope Creek Generating s Station. The purpose of this test was to demonstrate that the plant i response was in accordance with plant design for loss of all offsite

'

power scurces after the licensee had assured that the previous test ob-servations had been investigated and resolved. This loss of offsite power test was initiated with the reactor at cold (T <200 F) shutdown temperature and pressure conditions with the reactor mode switch in

, shutdos The significant observations during this test were: (1) the "B" safety auxiliary cooling system loop head tank level indicator failed; (2) one

\ control room emergency ventilation (air recirculation) system fan failed

'

to start; and (3) one drywell fan also failed to star Hope Creek station personnel observing the test identified a total of 17 condition '

Some of the 41 observations from these two tests did not have plant

, safely implications; nevertheless, it was the team's responsibility to:

(1) independently assess the root cause of each observation; (2) review the effectiveness of the corrective action planned or taken; and (3) assess the overall implications of the tes't result Each observa-tion was evaluated for its potential effect on systems not identified in the observation report . . . . - - - . - -

. a

, G

'

J A second cold loss of power test was conducted on October.2,1986. The -

Augmented Inspection. Team witnessed this test and assessed the result , One test observation was a repeat of a previous observation and involved j- a Bailey. logic module.

!

l The inspection began on September 25 and ended October 3, 1986.

Conclusions Of the 41 observations reported from the two loss of offsite power tests,

!

the overall safety significance was relatively minor except for the Bailey solid state logic module failures. Of eight hardware failures

' i dentified during this review, six were attributable to _ various malfunc-tions within the Bailey logic module Three weaknesses ~found with the Bailey logic modules were: (1) the depen- !

'.

dency on common equipment for accomplishment of automatic and manual safety actions for the actuated safety system equipment; (2) limited test provisions. to assure the online operability of the Bailey logic modules '

after their installation into the equipment cabinets; and (3) the use-fulness of the bench test equipment in assuring that the Bailey logic modules are operable. The team was also concerned that the failure rate

'of the Bailey logic modules appeared high. These weaknesses are espe-

-

l

'

cially significant since all of the balance of plant safety-related sys-tems (and a.part of one NSSS system) use Bailey modules to develop the safety system logic and actuation functions. Details of the team's con-

~

'

cerns on the Bailey logic module are contained in Section V of this

repor A number of miner plant design, construction, and manufacturing problems were also identified. Several specific weaknesses in the scope of vari-ous system preoperational tests were revealed since the loss of offsite

,

power tests were the first integrated demon'stration of the plant response

to this event. - Several subtle interactions involving the dependency of

~

various systems on cooling and instrument air supportin'g systems were ' revealed.

A number of observations resulted because instruments or other equipment lost power during the test. A number of these. instances involved the apparent failure to meet FSAR commitments to provide reliable power to

specific instruments or equipment.

III. ' Assessment of Results of the Loss of Offsite Power Tests i' An assessment of results for each reported observation from the hot loss of l power. test on September lith and the cold. loss of power test on September 19th was made by the Augmented Inspection Team. These results have been categor-ized by root cause as described below. The total list of observations is t

[

$

'l

- v-w , uma,,-e,,, r,n --.,--,---r.,,n-_,..v,,.,, --,e-r-. ,,_m-,.me.--,ve,,.,.g . , , - . , , , n,-,,.=-,-mm ----,,m.m.,m.----,, mew. , , - . , ,

C A

shown in Attachment When potential Technical Specification impact was identified by the team, the results of the team's assessment of that impact is provided in this repor Equipment Failures Of the 41 reported observations, eight (8) were equipment (hardware)

failures. Other than the Bailey logic module failures, there were no-significant individual hardware failures. A general discussion of Bailey logic module problems is also provided in Section . Service Water B Screen Wash Pump Hot Bearing Temperature Indication (H-11)*

The service water screen wash B pump developed a bearing high tem-perature alarm on the chronolog computer point A2493 starting at 6:26 p.m. on September 5, 1986. ~This alarm occurred because of un-intentional shorting of wiring inside the resistance temperature detec. tor to case ground following pump seal maintenance that same day. Frequent maintenance is required for the screen wash pump seals. This activity requires the removal and reinstallation of both temperature and vibration sensors. As discussed in Section III.4.1, this failure was not corrected prior to the September lith hot loss of power test. A false high bearing temperature alarm was provided since the B pump was operable, but was not actually operat-ing, at that time. The resistance temperature detector failure was caused by abrasion of wire conductor insulation as the detector cover cap was screwed onto its housing. No procedure exists for the installation and dressing of conductor leads for resistance temperature detector Technical Specification 3.7.1.2 requires two service water loops to be operable in Operational Conditions 1, 2, and 3, and one ser-vice water loop in Operational Conditions 4, 5, and when handling irradiated fuel in secondary containmen The limiting conditions of operation were not exceeded since the "B" loop screen wash pump remained operabl . Circulating Water System Pump and Valve Indications (H-12)

During the hot loss of power offsite test, erroneous status indica-tions were provided to the operator for circulating water valve

'

HV-2152D and pumps CP501 and DP501. Circulating water valve HV-21520 had concurrent indication of full-open, full-close, and mid-position as this valve was closing. The erroneous valve position indication was attributed to a sticking NAMCo valve limit ~ switch which was manually exercised and subsequently tested to confirm proper operation. Limit switch malfunctions have been observed during preoperational and surveillance tests suggesting that more frequent preventive maintenance may be necessary.

,

Licensee observation identification number, H = Hot; C = Col i

_ - _ _ . - _ _ . _ . . _ _ .

. _ _ _ , _ _ . , _ _ _ _ . . . , _ , _ _

G

Pump start-enable indicating lights on the main control board were not illuminated.even though both pumps were running during the tes These failures were caused by burned-out indicator. lamps whose high failure rate appears to be a chronic problem at Hope Cree . Emergency Diesel Generator C Output Breaker Failed to Automatically Close (H-24)

Emergency diesel generator C output breaker 52-40307 failed to auto-matically close to supply power to the 10A403 emergency bus. This

_

observation resulted from failure of logic input-number 7 to the Bailey solid state logic module used in the control circuit for main feed breaker 52-40308. This input failure was due to the failure of 2 Bailey logic modules as discussed in Section V. Hope Creek Technical Specification 3.8.1.1 requires the four sepa-rate and independent emergency diesel generators to be operabl When the C diesel breaker did not close, the C diesel was inoperabl The a.ction required was to start and load the remaining diesels and restore one offsite circuit within 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> Since the other diesels started and loaded properly and offsite power was restored in seven minutes, the applicable technical specification action state-ment requirements were satisfie . Drywell Recirculation Fan Failed to Start (C-1)

Drywell recirculation fan 1G1-VH212, which is in one of eight dry-well cooling units used to maintain drywell temperature and pressure conditions, failed to start during the cold loss of power test on September 19, 1986. Diesel generator load sequencer signals act through the Bailey solid state logic system to start the equipmen This same fan may also not have started during the hot loss of power test on September 11, 1986. At that time, test observers noted that fan ICl-VH212 did not start. However, no problems were subsequently found with this fan during the investigation. The process computer confirmed that at least one of the fans did not start on September 11, 1986; however, the process computer does not identify failures of individual fans, but rather the failure of one or more within a group of fan The failure was caused by a misaligned Bailey logic module connector within solid state logic panel 1AC653. The poor connection, which was not apparent during normal operation, interrupted an input sig-nal to the card between the printed circuit card and its mating connector. The absence of that signal prevented automatic startup of the fan. No work is known to have been performed on the 1G1-V212 fan control logic circuit that could have induced a failure at the printed circuit board connector between the two loss of offsite power test O e

A possible weakness during the investigation following the first loss of offsite power test is that after fan ICl-VH212 was found to be operable, the other remaining fans were not started as a con-firmatory test. Such confirmatory testing to determine the root cause of an observed failure may have revealed the problem with one of the fan . Process Start Inhibit Signal Did Not Inhibit Reactor Auxiliary Cooling System Pump Manual Start (C-2)

After completion of the cold loss of_offsite power test, reactor auxiliary cooling system pump BF209 was manually starte A process start inhibit signal should have prevented this action, but did not because of incorrectly positioned staple jumper 1 on a Bailey logic card. Jumper 1 was positioned for a 120VAC signal input instead of 24VDC, preventing the logic module from recognizing the presence of an inhibit signa Staple jumper 5 on the same module was also mispositioned. It would have caused failure of an indicator light to tell the operators that a bus power failure had occurre . Control Room Air Fan Failed to Start (C-7)

Control room return air fan BV415 failed to restart following the cold loss of offsite power test. This was one of two fans that return air from the control room to the supply system. A failed time delay module on Bailey card 4-10-14 in panel DC652 prevented the return air fan from startin Technical Specification Section 3.7.2 requires 2 operable control room emergency filtration subsystems, including return fans. With one return fan inoperable, the licensee was required to restore it to operability within seven days, which was don . 7.2KV Supply Breaker Timer Module Failure (C-13)

The alternate 7.2KV supply incoming breaker closed when offsite power was restored. This action occurred automatically without operator intervention. Upon a trip at the normal supply breaker, the operator is required to depress the flashing trip ligh This resets the memory function within the Bailey logic module to prevent automatic closure of the alternate breaker when offsite power is restored. Since the timer module failed in the flasher circuit for the normal supply breaker trip light, the trip indicating push but-ton did not flash and the operator did not reset the memory of the Bailey logic card. In the absence of this reset, the alternate supply breaker functioned in accordance with the design by auto-matically closing onto the bu ..

O 8 Automatic Breaker Close Input Buffer Failure (C-14)

The normal-incoming circuit breaker 52-10101 to bus 10A101 closed when the automatic close block was de-selected. After the loss of offsite power test was initiated, circuit breaker 10101 tripped on undervoltage. The trip light flashed as this was an uncommanded (i.e., not initiated by the operator) trip. The operator then acknowledged the trip and flashing of the trip light stopped. The Bailey logic module input buffer transmitting the undervoltage sig-nal failed (giving a false undervoltage signal) which caused the

~

breaker to automatically close when automatic close block was de-selected coincident with feeder voltage being restored. The mal-function could not be repeated on September 19. This particular failure was finally located ~and corrected when the same problem repeated during the second cold loss of power test on October 2, 1986 (see Section VI of this report).

B. Preoperational Test Deficiencies React'or Auxiliary Cooling Ssystem B Pump Did Not Automatically Start (H-7) '

The Hope Creek design provides an automatic start of the reactor auxiliary cooling system "B" pump only if the reactor auxiliary cooling system "A" pump fails to start during a loss of offsite pwer that is not coinc.ident with a loss of coolant accident. This particular observation during the hot loss of offsite power test reflected the actual design of the system. However, the team de-termined that this particular feature had never been fully tested during the preoperational or surveillance tests. Actuation of the

"A" pump was. tested, but actuation of the "B" pump in response to failure of the "A" pump to start was not tested. This was consi-dered to be a test program weaknes The team also noted that control room procedure OP-AB.ZZ-135(Q) was not clear in specifying this interlock, and this lack of clarity caused the observer to identify the reactor auxiliary cooling system

"B" pump failure to start as a concern. The procedure was subse-quently revised to add a note to the reactor auxiliary cooling sys-

,

tem puinp sequencing ste Subsequent to this procedure change, a design change was initiated to automatically start both reactor auxiliary cooling system pumps during a loss of offsite power conditio This design modification will be reviewed during a subsequent inspectio . Reactor Auxiliary Cooling System Pump Tripped Due to Low Expansion Tank Level (H-22)

The reactor auxiliary cooling water system pumps tripped during the hot loss of offsite power test because a low water level condition was reached in the system's expansion tank. The low water level s

.

..

e

condition resulted from a timing mismatch in stroke coordination between normally closed reactor auxiliary cooling isolation valves and normally open chilled water isolation valves that provide cool-ing water to the drywell cooling units. Chilled water normally supplies the drywell coole'rs. In a loss of power situation, the chilled water supply valves close and the reactor auxiliary cooling system supply valves open. Closure of the chilled water isolation valves and concurrent opening of the reactor auxiliary cooling sys-tem isolation valves was initiated by the diesel generator sequencer at 85 seconds after initiation of the loss of offsite power tes For approximately 59 seconds, the reactor auxiliary cooling system and chilled water systems were cross-connected such that reactor auxiliary cooling system water was diverted to the chilled water system. As a result, the 640 gallon reactor auxiliary cooling sys-tem expansion tank was pumped to its low water level at the same time that the chilled water 450 gallon expansion tank overflowe This deficiency was corrected by design change DCP 4-EMJ-86-962 that changed the start of closure for the chilled water system isolation valves from 85 to 13 seconds. These valves now ceach full closure

.before the reactor auxiliary cooling system isolation valves are opened. This design change was satisfactorily tested during the cold loss of offsite power test on Septenber 19, 198 The system cross-connection problem was not discovered earlier as the hot loss of offsite power test was the first time that automatic transfer of drywell cooler supply water occurred with both systems operating. The preoperational test had verified that the load sequencer caused the closure of the chilled water valves, the open-ing of the reactor auxiliary cooling system valves, and the starting of the reactor auxiliary cooling system "A" pump. However, during this preoperational test, manual flow balancing valves were closed prevented system cross-connections of the two systems. Therefore, complete operation of the reactor auxiliary cooling system during the period of valve transfer was not accomplished during surveil-lance-tests or the preoperational tes . Process Start Inhibit Function Did Not Inhibit Reactor Auxiliary Cooling System Pump Manual Start (C-2)

In addition to the incorrectly positioned Bailey module staple jum-pers previously described in Section III.A.5, an earlier preopera-tional loss of power test did not include steps to verify that the reactor auxiliary cooling system pump BP209 would not start whenever the process start inhibit signal was presen Because the'misposi-tioned staple jumpers were not detected prior to the cold loss of offsite power test, the control room operator was able to manually start this pum '

-. .- - . _ - . _ _ _ _ . - _ _ _ _ _ - . _- _aa -

__- 6 .m. . .

.

e

, -e

, Reactor. Building' Ventilation System Failure Due to Loss of

, Instrument Air (C-3)

The reactor building ventilation system started and maintained the required negative pressure (between' .2 and .3 inches of water gauge) in the reactor building for approximately 18 minutes.. At this point, the reactor building ventilation system dampers closed because instrument air pressure was being lost. Closure of the dampers caused the reactor building ventilation system fans to trip on low flow in accordance with the design. As described in the next (C-4) observation,. instrument air pressure was lost as a result of interaction between it and the reactor auxiliary cooling syste Performance of an integrated loss of instrument air test recommended in Regulatory Guide 1.68.3, "Preoperational Testing of Instrument Air and Control Air Systems", during the preoperational test program would have disclosed this reactor building ventilation system de-pendency'upon the instrument air and reactor auxiliary cooling sys-tem Specific operator actions should have been described in th. loss of offsite power test procedure to initiate reactor auxiliary cool-ing system flow so as to prevent a. loss of the instrument air syste . Emergency Air Compressor Trip (C-4)

i In a loss of power situation, cooling water is supplied to the emer-gency air compressors from the reactor auxiliary cooling system

,

through manipulation of manual valves. During the cold. loss of offsite power test, instrument air pressure prematurely decreased to the 85 psig initiation setpoint for the emergency air compresso This decrease was due to a partially opened valve that cross-con-nected instrument and service air systems.

. Because reactor auxiliary cooling system flow was not manually re-stored in a timely manner, the emergency air compressor started, but tripped out 9 seconds late Subsequent attempts by the control room operator to manually restart the emergency air compressor failed because of internal pressure sensed by the intercooler pres-sure switch. The compressor retained this internal pressure because l its protective trip did not initiate the normal pressure unloading sequence prior to shutdown of the compresso Each of these actions

was in accordance with the plant desig Approximately 20 minutes later, the reactor auxiliary cooling system

.

supply to the emergency air compressor was restored by the plant staff, and by that time the emergency air compressor internal pres-sure had decreased below the pressure switch setpoint. These two conditions satisfied the control interlock permissives needed for subsequent operation of the emergency air compresso '

_ _ ,_ . ._ _ _ _ . _ _ _ . _ _ _ . _ -,___ _. .

.

o

As stated previously, the reactor auxiliary cooling system /instru-ment air system interaction was not identified or tested during the

.preoperational test program. The team learned that emergency com-pressor operation had been demonstrated during the instrument air preoperational test, but at the time, compressor cooling was sup-plied by the fire protection syste An initial but incorrect determination was made regarding.the actual cause of failure of the emergency air compressor to restart. An equipment operator observed that the ground fault relay flag was actuated for the supply breaker of the emergency air compresso The relay flag was then locally reset, and the control room operator was able to manually restart the emergency air' compressor. However, it was subsequently determined that the electrical ground fault relay flag played no role in the successful restart of the_compres-sor as the relay flag had probably been actuated during preventive maintenance on the breaker on August 25, 1986, and may not have been reset at that tim The licensee modified the loss of instrument air procedure OP-AB.ZZ-131(Q) and planned to revise the instrument air system opera-tion procedure OP-SO.EB-001(Q). These' changes provide instructions-for manual unloading of the emergency air compressor whenever the automatic shutdown unload sequence has not been performed. The licensee also plans a modification to provide a compressor start inhibit whenever reactor auxiliary coolir,g' system cooling water is lost. This change will reduce the number of manual unload sequencas needed to be performed at the compressor. In addition, the licensee has documented specific operator actions needed to restore reactor auxiliary cooling system following a loss of offsite power in the abnormal operating procedure for this even Indeterminate and Unresolved Observations During the inspection at Hope Creek, the Augmented Inspection Team was not able to determine the specific root cause from the available evidence for the following failure . 4KV Bus 10A502 Supply Breaker Did Not Trip Open (H-9)

When the loss of offsite power test was initiated, a bus undervolt-age trip signal should have caused the normal incoming supply breaker 52-50201 to bus 10A502 to trip open. However, this breaker did not trip open on the undervoltage signal, and a subsequent re-mote trip signal initiated manually by the control room operator was also. ineffective. The breaker was then tripped locally. The purpose of the trip is to permit automatic transfers to be effected, as well as to preclude bus undervoltage conditions (<70% of normal bus voltage). Prior preoperational and surveillance tests had demonstrated satisfactory performance of the breaker, and the past

.- - , . __ _

.- _ _

. _ - _ _ . - - - .-

.

,.

maintenance history showed no similar problem. Preventive mainten-ance was last performed on the breaker in November 1985 on a 36 mooth interval. Troubleshooting was performed on the breaker which included racking out of the breaker for physical inspection and an operational verification in a test cell. Simulated bus undervoltage and loss of offsite power conditions for this breaker did not dupli-cate the observed failure. The DC control power for the breaker was instrumented and no discrepancies were noted. During the cold loss of offsite power test, proper operation of the breaker was veri fied. On 9/29/86, the licensee performed tests on the breaker's Bailey logic control module in accordance with procedure IC-GP.ZZ-031. No equipment performance problems were identified. The lic-ensee then instrumented the control signals to the breaker prior to the conduct of cold loss-of offsite power on 10/2/86 to monitor breaker performance, but no failures were noted during this tes . Safety Auxiliary Cooling System Pump 8 Did Not Start (H-10)

An observer noted that the safety auxiliary cooling system pump B did not start during the hot loss of offsite power test. This problem did not recur during subsequent diagnostic testing, and monitoring instrumentation installed for the cold ' loss of power test confirmed proper starting of this pump. Preoperational test records indicated that the pump had operated properly with the sequencer in accordance with the plant design. An unrelated Bailey module failure, as described in section V.A.1, also'could not have caused the observed safety auxiliary cooling system pump failur . Control Room Ventilation Damper Failed to Close (C-8)

Two redundant isolation dampers are provided in each of two control room inlet air supply lines and in the common control room exhaust line. These dampers isolate the control room when high radiation is detected in the ventilation inlet or on a loss of coolant acci-dent signal. Normally energized solenoid valves control instrument air to open the isolation dampers against mechanical spring pressur The solenoid valve for the outside isolation damper in each inlet line and for both dampers in the exhaust line is connected to an uninterruptible power source; hence, these four dampers should not close during a loss of offsite power condition. The solenoid valve for the inside isolation damper in each inlet line is connected to an emergency diesel generator power source which causes these two dampers to close on a loss of offsite power and then reopen as each individual ventilation system start During the cold loss of offsite power test, an observer noted that the control room indicator lamp for isolation damper HD9588AA showed that this damper had not closed. This damper is connected to the diesel bus. Damper HD9588AA was subsequently cycled several times and performed satisfactorily. Subsequent investigation suggests

O o

that the observation may have resulted from a system interaction involving the loss of instrument air to the damper solenoid valve Since the observed failure could not be repeated, the licensee has committed to cycle the damper at least once every 31 days to verify its operabilit D. Observation Error and Other Non-Problems Main Steam Relief Valves H and P Actuation (H-1)-

Main steam relief valves H and P are specified to_open on increasing reactor. pressure at 1047 psig to provide steam pressure relief to the suppression pool. A test observer noted that relief valve P did not open even though relief valve H had opened during the hot loss of offsite pressure tes At the time that relief valve H opened, reactor pressure was increasing at a very slow rate. The licensee subsequently examined the most recent calibration records for the pressure transmitters that control the opening of relief valves H and It was determined that the setpoint for relief valve P was approximately 4.5 psig higher than the value established for relief valve H. Each setpoint was within its permitted toler-ance of approximately plus or minus 6 psig. Consequently, while the observation during the test was correct, the main steam relief valves did operate properly in accordance with the plant desig . 120VAC Uninterruptible Power Supply Trouble Annunciator Reset (H-3)

Prior to the hot loss of offsite power test, maintenance was being performed on one 120VAC uninterruptible power suppl The electri-cal loads normally connected to the out-of-service uninterruptible power supply were being supplied by another uninterruptible power supply. The connection of these additional loads is accomplished by manual transfer switch 10N401. Uninterruptible power supply trouble annunciator D3-E3 began to flash during the loss of offsite power test, and remained illuminated when the control room operator tried to reset it as the diesel generators were' connected to the emergency buse The observer noted that the annunciator remained illuminated despite attempts by the control room operator to clear the alarm. The licensee determined that the annunciator alarm was a correct indication since manual transfer switch 10N401 was not in its normal (i.e., each uninterruptible power supply in service)

position. Hence, the uninterruptible power supply trouble annunci-ator performed correctly in accordance with design. The team de-termined that the Hope Creek training simulator assumes that the transfer switch is always in its normal position; hence, an off-normal condition has not been provided in the training situatio a o

14 Hydrogen Recombiner Alarm (H-5)

After offsite power was restored on September 11, individuals in the control room objected to the noise level of the hydrogen recom-biner alarm, and its volume level was then lowered by adjustment of the alarm horn located in panel 633. This alarm met the design intent, but is duplicative of a main control room annunciator for the hydrogen recombiner. The scope of a recent control room human factors review in accordance with NUREG-0700 guidelines did not cover audible alarms originating outside of the front and back panels in the control roo Hope Creek preoperational and surveil-lance procedures also do not contain criteria for the voluma loud-ness of horn alarm . Drywell Recirculation Fan Failed to Start (H-13)

This particular test anomaly, which was recorded during the hot loss of offsite power test, appears to be an observation error involving the i.dentification number of a drywell recirculation fan that failed to star The failed fan was identified as ICl-VH212 on September 11, but subsequent testing of this fan could not duplicate the failure. As described in sections III.A.4 and V.A.3, drywell re-circulation fan 1G1-VH212 failed to start during the cold loss of offsite power test due to a misaligned Bailey logic card connecto The team believes that the wrong identification number was recorded during the first test since the manual controls for these two fans are adjacent to one another on the main control room panel and a defect was found for fan 1G1-VH212, 5. Main Steam Relief Valve P Acoustic Monitor Valve Position Indication (H-15)

During channel testing prior to the hot loss of offsite power test, the acoustic monitor for main steam line relief valve P was found to be inoperativ The defective sensor remained connected to its control room display, but the display channel for relief valve P was declared to be out-of-service. During the hot loss of offsite power test, an observer noted that whenever main steam relief valves H or K were opened, the acoustic monitor display for relief valve P would also indicate an open status even though the P relief valve was closed. The defective sensor was subsequently disassembled, and an internal mounting screw holding two crystals in place was found to be loose. A second acoustic monitor sensor was found with similar symptoms, and both units were returned to the vendor for detailed failure analysi It is believed that mechanical vibration, which occurred whenever the H or K relief valves were opened, caused the two crystals in acoustic monitor P to vibrate and produce a false signal on the display for relief valve P. Other indications

.

15 such as steam relief valve discharge pipe temperature and steam relief valve solenoid valve position confirmed that relief valve P remained closed throughout the tes . 6. ~ Rod Sequence Control System Display Was Lost (H-21)

An observer reported that the main control room display for the rod sequence control system was lost during the hot loss of offsite power tes This system augments procedural controls for the selection and movement of control rods within the reactor during startup and shutdown activities. The rod sequence control system logic is powered by uninterruptible buses; however, its display panel uses non-Class IE interruptible power. The observed loss of this display was correct, and is consistent with the design intent of this system. The loss of offsite power test procedure was re-vised to state that loss of rod sequence control system display is anticipated for this even . Loss of Instrument Air (H-23)

A loss of instrument air was observed during the test. As described in section III.B.5, the loss of instrument air was caused by auto-matic tripping of the emergency air compressor after 9 seconds of operation because of a lack of reactor auxiliary cooling system water flow. The observed loss of instrument air was correct and was consistent with the plant desig . Service Water Yard Dump Valves Opened (C-9)

For approximately one minute, the service water yard dump valves opened during the loss of offsite power test and dumped water into the' yard area. An observer noted that the valves were open, and an operator visually confirmed that the dump had occurred and that the valves had automatically reclosed. Opening of the yard dump valves is required to remove entrapped air from service water system piping, and the system operated in accordance with its design inten The licensee subsequently performed a check of the calibration of pressure transmitters used to control opening and closing of the yard dump valves, and confirmed that the setpoints wore prope However, the licensee is considering the feasibility of adding a time delay in the electrical circuitry to min'imize future spurious yard dump actuation . Control Room Ventilation Heaters Were Dusty (C-12)

During the cold loss of offsite power test, both control room ventilation air handling units and chillers were operating. This amo' int of cooling capacity caused the control room temperature to drop to the heating thermostat setpoint. An accumulation of dust on the electric heating coils caused a burning odor to be noticed

_

.

N

throughout the control room. The licensee subsequently inspected the control room air handling and heating units and found them to be relatively clean and free from any apparent safety hazard . Alarms to Chronology Message on CRT Screen (C-17)

Each control room cathode ray tube display provides up to six lines of alarm messages for the most recent off-normal and return-to-normal conditions that might be of immediate interest to the control room operator In addition to the current six line display, the system permits the operator to scan 40 additional "pages" of six line displays containing past alarm conditions. The Hope Creek chronolog computer system receives each incoming alarm from the plant, stores the information in memory, and generates the data presented to the operator in the six line CRT display. During periods of high alarm activity, the chronolog system does not at-tempt to maintain the six line CRT display on a "real-time" basis, and alerts the operator ~that the CRT display is not current. This alert is provided in the form of a CRT message stating that current alarms have been directed to the chronolog syste During the cold loss of offsite power test, the chronolog system received a large number of alarms and provided the " Alarms to Chronolog" message to the control room operators. A test observer noted that this had occurred; however, this response was normal and met the design intent of the information syste E. Design and Desig'n Control Is:ues Turbine Building Chilled Water Chillers (H-17)

Several problems were observed in restarting the turbine building chilled water system after the hot loss of offsite power test was terminated. One of these problems represented a design weakness '

in vendor supplied equipmen Two of four chiller discharge valves had concurrent full-open and full-closed indications. These normally closed valves automatically open during a loss of offsite power condition and then return to their normally closed position when offsite power is restore However, the two valves could not fully reclose because of the loss of instrument air and stopped at a mid-travel position. When the valves are in a mid-travel position, both the full-open and full-closed lights are illuminated. Therefore, the observation was cor-rect and the system performed in accordance with the design inten System operating and abnormal operation procedures did not explicitly identify the dependence of the turbine chilled water system on the instrument air system as a required supporting syst3 .

a

The AK111 chiller started, but tripped out because turbine auxiliary cooling system flow was not available. This action met the design intent; however, the licensee subsequently revised the chilled water system operation procedure (0P-50.GB-001(Q)) and abnormal operation-procedure (OP-AB.ZZ-135(Q)) to explicitly state that turbine auxiliary cooling water is required for operation of the chiller The DK111 chiller started, but would not load. The licensee deter-

'

mined that two setscrews set 90 degrees apart in a chain drive pulley were loose which permitted the pulley to rotate without moving the guide vane shaft. The vendor's design developed in 1975 provided only friction contact between the point of the setscrews and the round shaft of the. guide valves. Subsequently, the vendor provided details of a 1980 design modification where the' setscrews could be recessed into small holes drilled in the round shaft and

<

secured with a compoun . Source Range Monitor / Intermediate Range Monitor Detectors Would Not Drive Into the Core (H-19)

Prior to the hot loss of offsite power tests, the 4 source range and 8 intermediate range neutron monitoring system detectors had been withdrawn from the reactor core to their normal storage loca-tion below the reactor. Following the automatic reactor scram in-itiated by the loss of offsite power, these detectors should be in-serted into the reactor core by the operator to permit measurements of core power below 1 to 3 percent of rated power. When.the opera-tor attempted to reinsert the detectors, the drive mechanisms did not operate because they were connected to a non-1E interruptible power source. In its FSAR licensing submittal for Regulatory Guide 1.97, the licensee committed to providing a reliable power supply in accordance with qualification category 2 rather than Class 1E power sources as stated in the guide. Reliable power supplies were provided for the electronic equipment in the source range monitor and the intermediate range monitor channels, but no change was made to the original power sources used for the drive mechanisms. Since the Hope Creek simulator does not accurately reflect the present design, the operators were unaware that the loss of offsite power would prevent insertion of the detectors into the reactor cor . Reactor Building Supply and Exhaust Fans Did Not Start (H-20)

,

The reactor building supply and exhaust fans did not operate in

> accordance with FSAR Section 9.4.2.2.7 which stipulates that the fans automatically restart and continue to operate on standby diesel power. The fans started in a timed sequence as required, but then

,

tripped on low flow because the reactor building ventilation system

supply and exhaust fan dampers failed to open. In December 1985, i a startup test engineer identified that the dampers for the reactor building ventilation system supply and exhaust fans were supplied

,

- . . - -- _ . . , - . .- .-.- _- . . . - _ . - - - - - - . - - - - . - . ,

.

from non-Class IE power sources. A design change package / design

. change request package was issued and materials were available on-site to support implementation of the power supply change by May 23, 1986; however, systems engineering determined on June 6 that retest requirements were needed and on August 8 the offsite safety review group identified other issues with the design change package that required resolutio Neither the systems engineers nor the offsite safety review group identified that the plant was being operated differently than the design basis description in FSAR Section 9.4.2. The design change documentation did not identify the FSAR licensing commitment. De-sign control weaknesses were. evident in that the FSAR commitment was not addressed in the design change request / design change package safety evaluation, and as a result a 10 CFR 50.59 safety evaluation was not performed to allow operation of the facility differently than described in the FSAR. The system operational readiness review conducted by the designated system engineer on April 10, 1986 in accordance with system engineering memorandum 0010 failed to iden-tify the FSAR commitmen Open design change packages were reviewed to insure that no modifications are required to be implemented prior to plant restart because of improper prioritization. Design change package No. 867 was implemented on September 17, 1986 and was satisfactorily tested during the September 19, 1986 cold loss of offsite power tes . Safety Relief Valve Full Open position Indication (H-21)

Normally closed main steam line safety relief valves open when reactor pressure increases above prescribed setpoint values to pro-vide steam pressure. relief to the suppression pool. During the hot loss of offsite power test, control room observers noted that the

. safety relief valve position indicating system (acoustic monitors)

showed that all fourteen safety relief valves were open when most, if not all, were actually closed. These erroneous indications occurred because the acoustic monitors were supplied from a non-Class IE interruptible power source rather than the uninterruptible power source commitments for Regulatory Guide 1.97 category 2 equipment as stated in the FSAR. Preoperational tests were not performed in a manner that would have disclosed this particular de-sign error. Design change package 4-EME-86-963, implemented after-the September 11, 1986 test, corrected this design erro The team determined that Technical Specification requirements for the safety relief valve position indication system were not met prior to the implementation of the design change. The minimum number of operable indication channels were not available for loss of offsite power event .

.

19 Reactor Auxiliary Cooling System Pump Tripped Due to Low Expansion Tank Level (H-22)

.As described in section III.B.2, the reactor auxiliary cooling system and chilled water systems were cross-connected for approxi-mately 59 seconds due to improper valve coordination between these systems. As a result, water in the reactor auxiliary cooling system expansion tank was pumped to its low water level at the same time that the chilled water system expansion tank overflowed. The ori-ginal design for these systems did not consider the effects of sys-tem cross-connection Design change package 4-EMJ-86-962 changed the sequencer time for initiation of chilled water isolation valve closure from 85 to 13 seconds so. that these valves would be fully closed prior to opening of the reactor' auxiliary cooling system isolation valves. The implementation of this design change was demonstrated during the cold loss of offsite power tes . Safety Auxiliary Cooling System Loop B Head Tank Level Indication (C-6)_

A post-test-systems performance review of control room information display system records indicated that the safety auxiliary cooling system loop B head tank was empty when it was not. This erroneous indication occurred because the head tank level instrument was supplied from a non-Class 1E interruptible power source, and pro-duced a zero percent tank water level output signal when offsite power was lost. The corresponding head tank level for safety auxiliary cooling system loop A was supplied from a reliable power source, and remained operable during the loss of offsite power tes 'As a result, ambiguous indications regarding the status of the safety auxiliary cooling system were provided to the control room operator. The difference in instrument power selections between redundant safety-related cooling loops did not satisfy either operational'needs or human factors criteri Preoperational and surveillance tests did not disclose this difference in the design configuration. The licensee has initiated a design change (design change request 4-HME-86-1262) to place the safety auxiliary cooling system B head tank level instrument on a reliable bu Furthermore, the same condition should have occurred during the hot loss of offsite power test, but was apparently not identified at that tim . Service Water Loop B High Pressure Alarm (C-10)

An alarm generated during the cold loss of offsite power test indi-cated that high pressure existed within loop B of the service water system when it did no The corresponding instrument in service water loop A did not alarm. For the same reasons described in section III.E.6, this observation occurred because the instrument

.

in loop B was supplied by a non-Class 1E interruptible power source whereas the corresponding instrument in loop A was connected to a reliable power source. Ambiguous indications of the status of the service water loops were provided to the operator'in this situatio Preoperational and surveillance tests did not disclose this differ-ence in the design configuration. The licensee is planning to use consistent power sources for non-1E instruments in the redundant service water loop . Reactor-Vessel Water Level Indicators Lost Power (C-11)

Reactor vessel water level indicators LI-R604-B21 (-150 to +60 inches) and LI-R605-821 (0 to 400 inches) located in the main con-trol room lost electrical power during the test. As indicated in Table 7.5-1 of the FSAR, LI-R604-B21 was connected to the reactor protection system motor generator set, and loss of this indicator was in accordance with the design. However, this table stated that LI-R605-821 would be supplied from an uninterruptible power source, but the implemented design did not fulfill this commitment. The licensee subsequently developed a design change (design change request 4-EME-86-976) to provide a reliable power source for this indicator. Preoperational testing did not include a loss of power to the non-Class 1E buses; hence, this design inadequacy was not evident. The same condition should have occurred during the hot loss of offsite power tests, but was_not detected by the test ob-server During the inspection, the licensee.and architect engineer performed a review of FSAR Table 7.5-1 power source commitments. Several other control room instruments were found to be on non-Class 1E interruptible power rather than on uninterruptible sources. In two instances involving-condenser vacuum (PR-1664A, B, and C) and rad-waste tank level (LR-R008 and LR-R024), the FSAR will be modified to delete the uninterruptible power supply commitment as these are Regulatory Guide 1.97 Category 3 ' measurement In the third in-stance, the residual heat removal heat exchanger outlet temperature recorder (TR-R605-E11) will be connected to an uninterruptible power supply source by design change request 4-EME-86-96 . Reactor Auxiliary Cooling System Pump Motor Current Indication Was Lost (C-16)

During the cold loss of offsite power test, an observer noted that control room ammeters did not indicate the electrical current load for the reactor auxiliary coolant system pump motors. The loss of this indication did not affect the operation of reactor auxiliary coolant system equipment, and in this instance, loss of the ammeters was in accordance with the plant design. Ordinarily, it is a good design practice to power monitoring instrumentation from the same sources used to operate the ecuipment being monitored; however, the

.

existing design used space heater power supplies located in the motor control centers for the 120VAC power input to the curren transducer for the ammeter. This space heater supply is not avail-able during a loss of offsite power. This item is a design weakness from a human factors standpoint. In addition, the simulator did not reflect the as-built design of the plant in this regar This condition should also have been observed during the hot loss of offsite. power tes F. Procedure Adequacy Issues Reactor Core Isolation Cooling System Pressure Control to Water Level Control Mode (H-6)

During the hot loss of offsite power' test, it was noted that operat-ing procedure OP-SO.DB-001C(Q) did not describe the sequence of operations needed to switch the reactor core isolation cooling sys-tem from its pressure control mode (recirculation to and'from the condensate storage tank) to its water level control mode (direct injection into the reactor vessel) and vice vers During system preoperational tests, this particular procedure had not been use The control room operators were able to effect the mode transfer using-an "on-the-spot" step by step pro'cedural change. These in-structions will be formally incorporated into the procedure as ' art p

of the licensee's long term corrective actio . Reactor Building Filtration Recirculation and Ventilation System Failed to Automatically Start (H-14)

The reactor building filtration recirculation and ventilation system is actuated by an accident signal or high reactor building radiation to remove airborne radioactive material from the reactor building atmospher The filtration recirculation and ventilation system is not required to start for a loss of offsite power conditio The original plant design did, however, cause the initiation of filtration recirculation and ventilation system because of a subtle and unintended interaction with the reactor protection system. -When power was-lost, the reactor protection system motor generator sets caused an automatic reactor scram by de-energizing reactor water level, drywell pressure, and other trip channel This action caused an automatic start of the filtration recirculation and ven-tilation system which was not required (unless the loss of offsite power was also coincident with a loss of coolant accident). A de-sign change (design change package 358) was implemented in January 1986 to add a core spray initiation signal for on reactor water level as a confirmatory logic input. Since the core spray reactor water level and drywell pressure channels are normally energized

~

by DC battery sources, they are insensitive to a loss of offsite power conditio .

..

The abnormal condition procedure used during the hot loss of offsite power test was incorrect since it did not reflect the latest plant design; consequently, the test observers noted that the filtration recirculation and ventilation system failed to start during the loss of offsite power tes Revision 2 of procedure OP-AB.ZZ-135(Q) had deleted a note s_tating that the filtration recirculation and venti-lation system was initiated only by a loss of coolant acciden For this test, the filtration recirculation and ventilation syste did not start which was in accordance with the latest design. How-ever,.there are apparent weaknesses in the revision of plant proce-dures in response to plant design changes since the loss of offsite power abnormal test procedure was not current and training lesson plan 42 did not. reflect the current design. In addition, the team noted the list used to track differences between the simulator and the plant did not contain the design change package 358 modificatio . Emergency Diesel Generators Shifted to Droop Mode (H-16)

The emergency diesel engine governor is provided with an alternate speed droop (load control) mode for stable operation in parallel with other power sources; when onsite buses are energized indepen-dent of offsite power, the governors operate in the isochronous (frequency control ) mode. When the speed droop mode is manually selected by a main control board push button, a memory circuit is energized through relay seal-in contacts. Although the control circuit will automatically transfer to isochronous when speed droop conditions are not met, the circuit is reset only by selecting iso-chronous on adjacent control push button. The governor will operate in isochronous any time that 13.8KV power is lost to feeder buses, and it will then return to spend droop after 13.8KV is restored if the memory circuit has not beed reset. Control board push buttons in'dicate the actual mode in which the governor is operatin Licensee personnel observing the hot loss of offsite power test noted that the A and B emergency diesel generator governors auto-matically shifted from isochronous to speed droop mode when offsite power was restored through 13.8KV feeder buses. This was recorded as a test observation, but was subsequently found to be in' accord-ance with design. The governors were probably never reset from droop to isochronous when the diesel generators were shut down dur-ing their last operatio Although it has been the normal practice for the control room operators to reset the governor mode, it was not required by procedur The licensee has revised the emergency diesel generator procedures

,

to require that the operators return the governor mode to iso-chronous when securing the machines. In addition, the licensee has issued a design change request (86-1250) to ::onsider a change such that the governor control circuit will be in droop only when manually selected.

i

. _ - - - - - -, . - - - - . . _ . . . . - - . - - . - - . _ - - - --

.

The simulator accurately reflects the plant design in both governor mode transfer and indication. Training Lesson Plan No. 68 also addresses the operation of the mode select' circui The automatic operation of the governor mode circuit was identified as a possible problem in a startup deviation report KJ-1416 written against the emergency diesel generator preoperational test (PTP-883).

It was dispositioned on February 4, 1986 as a proper design. The licensee did not implement any procedural improvements as a result of this deviation repor . Turbine Building Chilled Water Chillers (H-17)

As described in section III.E.1, chiller AK111 started after the hot loss of offsite power test was terminated, but tripped out be-cause turbine auxiliary cooling flow was not available. In addition, two chiller discharge valves stopped in mid-stroke when instrument air was lost. The licensee has revised the chilled water system operating procedure OP-SO.GB-001(Q) and the loss of offsite power abnormal operating procedure OP-AB.ZZ-135(Q) to explicitly state that turbine auxiliary cooling water is required for operation of the chiller. units. While these two procedures contain an implicit recognition of the need for instrument air, the procedures do not explicitly state that instrument air is a required supporting system for proper operation of the chiller discharge valve G. Construction and Manufacturing Issues Diesel ~ Area Heating Ventilation and Air Conditioning Equipment Fire Alarm (H-8)

A fire alarm was indicated in the diesel area heating ventilation and air conditioning equipment room 5704 during the hot loss of-offsite power test; however, no fire actually existed. The fire computer system signalled the alarm from one or more of 20 detectors located at elevation 178 of the diesel generator building. This alarm occurred approximately one minute after the start of the loss of offsite power test. Because of an unrelated wiring reversal error at the multiplexer termination board, the alarm was initially identified as being at elevation 163 rather than 17 The alarm is believed to have resulted from diesel exhaust fumes leaking past annulus seals at the diesel exhaust stack Diesel exhaust was first noticed within the building during tests conducted in March 1986, and a plant modification (design change package 4-ECC-86-469) is scheduled for completion in late October. This modification will alter the diesel exhaust hoods so that exhaust fumes will not reenter the diesel generator buildin .

.

24 Control Rod Display Full ~in Lights (H-18)

Seventeen control rods did not provide a full-in status light on the full core display after the hot loss of offsite power test was

,

initiate The operator then used the four rod display on the main control board to confirm that all 17 control rods were fully in-serted into the reactor core. Because of the physical location of reed switches within the control rod drive assembly, a difference of approximately 0.25 inches exists between the position "00" reed switch and the " full-in" reed switch. The 17 control rods had reached the "00" position, but did not indicate the " full-in" posi-tion. This has been attributed to reduced pressure on the bottom piston of the drive because the control rod drive pump was not operating at the time. This effect has been observed at other BWR plants (e.g., Limerick), and a design modification had previously been developed to eliminate any ambiguity. With this modification, actuation of either the "00" or " full-in" reed switch will cause the full-in light to be illuminate The design modification required the replacement of two programmable read-only memory chips on one printed circuit card in the rod post-tion indication / reactor manual control system. The original chips carried an "A" suffix designator, and the replacement chips were identified by a "B" suffi In this instance, the manufacturer replaced one "A" chip with the new "B" chip, but failed to replace the second "A" chip. This manufacturing error was not detected at the factory or during receipt, reinstallation, or checkout of the

~

printed circuit board onsite. The team noted that the control rod position indication was not considered safety-related, thus 10 CFR 50, Appendix B controls were not required. Public Service Electric and Gas has issued a corrective action request (HS-86-C024-0) to the vendor for its respons . Emergency 01esel Generator C Speed Changer Did Not Respond to Control Room Controls (C-15)

During the cold loss of offsite power test, the control room opera-

-

tor noted that the C emergency diesel generator speed changer did not respond to an initial demand for a change in generator frequenc ! This occurred as the operator was trying to parallel the diesel generator with offsite power to transfer electrical loads. The defect appeared to be intermittent, and corrected itself after several attempts to adjust frequenc The speed changer is used to vary either generator frequency or electrical load, and must be operable for any manual diesel genera-tor operations, s

,, , , _ _ _ , . _ _ _ . , _ _ _ _ _ _ . _ _ _ _ _ _ _ . . . _

_ _ . _ _ _ . - _ . . - - __ ___

.

The licensee determined that the cause of this problem was a loose bezel connector on the RZ module in the main control board. The connector was tightened and the diesel generator tested satisfac-toril This problem may have existed since the construction in-stallation of the cables since there were no previously completed work orders for this area of the main control room boards. The team was informed that previous testing had not identified this proble The licensee has since systematically checked the tightness of other control board connector H. Operator Error Service Water B Screen Wash Pump Hot Bearing Temperature Indication (H-11)

As described in section III.A.1, a high temperature alarm was generated for the service water 8 screen wash pump bearing on the chronolog and CRT. The alarm was caused by shorting of one resis-tance temperature detector lead to case ground, and was recorded by the computer at approximately 6:26 p.m. on September 5, 198 The failed sensor provided an alarm that was observed during the hot loss of offsite power test on September 11, 1986, and a work order was initiated to correct the wiring detect the following da However, no work order had been prepared prior to initiation of the hot loss of offsite power test in response to the alar I. Security Diesel Fire Pump Room Door Lock (H-4)

On September 10, 1986 when fire protection personnel entered the diesel fire pump room to conduct a weekly operability test, the door keylock worked properly. As test observers entered the area shortly before the hot loss of offsite power test began, the door keylock would not work. The cause was determined to be a faulty lock cylin-der which was then removed from the doo No door lock is actually required within the protected area that includes the diesel fire pump room according to the licensee. The lock was only provided for optional industrial security reasons. The licensee intends to keep the door unlocke J. Training Reactor Core Isolation Cooling System Minimum Flow Valve Did Not Open (H-2)

When the reactor core isolation cooling system was returned from water level control (direct injection into the reactor vessel) to pressure control (recirculation to and from the condensate storage tank) by the control room operator, an observer noted that the

, .

.

!.

reactor core isolation cooling system minimum flow valve did not open. The control logic requirements for opening of reactor core isolation cooling valve 1-BD-SV-F019 are that the system flow be below 90 gallons per minute and the cooling pump discharge pressure

,

be greater than 125 psi The licensee examined the shift log and control room'information display system data taken during the hot loss of offsite power tes This review indicated that the change from water level control to pressure control occurred at 2043 according to the shift log. This operation required the control room operator to close injection valve 1-BD-HV-F013 and open the pump test return valve 1-BD-HV-F02 An observer reported seeing the operator open valve F022 just as valve F013 indicated closed; valve closure indication occurs while the valve is still open by 2 percent in the closing directio <

Prior to 20:44:36, the control room information display system in-dicated that the reactor core isolation cooling system was at full flow injecting into the vessel. At 20:44:36, a control room infor-matio.n display system alarm indicated that the system flow was at 250 gpm. At 20:45:06, the flow was at 480 gpm. From this. data, the licensee concluded that the reactor core isolation cooling sys-tem flow never dropped to the 90 gpm flow value needed to satisfy the minimum flow valve control logic. A subsequent confirmatory test demonstrated that valve 1-BD-HV-F019 was operable and would open at the 90 gpm and 125 psig control logic conditions. On this basis, no equipment failure occurred, and the observation, while correct, did not properly take into account the control logic con-straints imposed on opening of the minimum flow valve. This obser- .

vation implies that a minor weakness in training may exist regarding the control of the minimum flow valv . Service Air Isolation Valve Had Dual Indication (C-5)

Concurrent indication of full-open and full-closed was observed in

!

the control room for service air isolation valve HV-7595 after the cold loss of offsite power test. This valve should have been in a closed position to isolate service air loads from the instrument air system; however, it was subsequently determined that a mechani-cal handwheel on the valve had been turned from its normal (i.e.,

valve fully closed) position to a point where the valve could not fully clos Preoperational testing had previously verified the proper stroke and operation of this valve. There appeared to be a minor training deficiency regarding plant staff knowledge of the operation of this valv Subsequently, the licensee conducted specific personnel training regarding the proper valve lineup for this system and the means used to identify when the handwheel is in its proper position.

.

- - . -- . - _ , . - . - . . - . - - - - _ . - , . , . - . . . . -. - . - . .

r

,

IV. Other Investigative Efforts During the inspection, the team examined previous events and problems that had occurred at Hope Creek or other plants. The purpose of this review was to identify whether common root causes existed among these events and those identified during the loss of power test Other Related Events at Hope Creek Generating Station Diesel Generator "A" Failure to Start (6/10/86)

The "A" emergency diesel generator failed to successfully complete its starting sequence on June 10, 1986. This event was reviewed by the inspection team to determine if it reflected on the thorough-ness of preoperational testing or if it was a precursor to problems observed during loss of offsite power testing on September 11 and 19, 198 The diesel generator was started as part of an investigation into a previous spurious actuation of the standby liquid control system (Licensee Event Report 50-334/86-028). After. initial acceleration, the diesel governor operated erratically causing speed oscillations and shutdown of the diesel engine within twenty (20) seconds. The licensee found a defective motor-driven potentiometer associated with the generator voltage regulator. The defective item was re-placed, and the diesel generator was returned to servic The electric governor determines frequency from the generator static exciter output which is applied to saturable reactors in its error sensing circuit. Because of their operating characteristics, large changes in voltage are also sensed by the saturable reactors as an error in frequency. During this event, the motor-operated potentio-meter failed causing the automatic voltage regulator to make com-pensating corrections. However, the potentiometer failure caused a voltage error which was beyond the effective range of the auto-matic regulator. The voltage error was interpreted by the governor system as a speed error and caused the governor to produce frequency oscillations between the mechanical speed setpoint of 62Hz and a low frequency value indicative of minimum fuel supply. This re-sulted in excessive piston movement within the governor which ported a greater capacity of oil than its internal pump could supply. The governor servo piston returned to a minimum fuel position and shut the fuel racks. Since the diesel generator shutdown occurred be-cause of a low fuel rate, no diesel trips were recorded and only

" generator under frequency" and " low fuel oil pressure" alarms occurre _

r

..

.

This event was caused by the failure of the manual voltage regulator motor-operated potentiomete The inspection team found that fail-ure does not reflect on the quality of the preoperational test pro-gram. Also, this failure was not connected with any failure ob-served during loss of offsite power testin . Primary Containment Drywell Temperature Problems During initial plant operations, the licensee became aware of high temperature conditions within the primary containment drywell. The most significant problems were above the reactor head insulation package, below the refueling cavity seal plate, and at the top of the biological shield air gap. There was also a high differential temperature between the reactor vessel support skirt and the vesse These conditions were examined by the inspection team to determine if it reflected on the thoroughness of preoperational testing or on design contro The preoperational test program cannot realistically identify in-sulation effectiveness problems because the necessary high system temperatures are not achieved. However, it does verify the oper-ability of drywell cooling and temperature monitoring system There were no problems identified by the. team which reflected on the conduct of the preoperational test program in this regar The causes of the high temperatures were found to be incomplete application of insulation, insufficient ventilation return air flow, and improper placement of temperature elements. Station system engineers were assigned to investigate and correct these problems during the Startup Test program. The inspection team found that these individuals were knowledgeable and had addressed the issues after conducting extensive inspections within the drywell. Correc-tive actions included measures to increase the efficiency of in-sulation, modifications to correct ventilation and air circulation flows, and relocation of temperature elements to have representative temperature locations. Infrared cameras and pyrometers were used for survey The team was briefed in detail as to the problems identified and

. corrective actions taken. Plant design drawings were reviewed and inspections were conducted within the primary containment drywell for each problem. Although additional insulation was applied to correct some problems, there were no major changes that would indi-cate poor original design control. This was also true for the ven-tilation system changes and the relocation of temperature element Reactor Vessel Head Area The licensee found small gaps in the reactor vessel head insu-lation package around the three vessel head nozzles. Insula-tion was added to seal these gaps. A temperature sensing

e k

,

thermocouple was turned away from an uninsulated check valve on the vessel head spray line. Insulation was applied to the check valve as well as the reactor head vent (to sain steam)

line. Ventilation supply was. improved by; removing a restric-tive "T" duct from supply lires above the refuel cavity seal plat Refueling Bellows Area The original plant design provided a ventilation ' return (ex-haust) from below the refueling bellows seal area with inlets on the top and the bottom of that duct. All bottom returns were blanked and relocated to the top of the duct in an attempt to remove more hot air from the upper elevations.~ The refuel-ing cavity bellows skirt was insulated, as were the main steam lines at high energy pipe rupture restraint contact area Temperature sensing thermocouples were relocated away from hot pipes. In each case, the final location was close to the original position and therefore met the the design specifica-tions for the instruments. Additionally, insulation was ap-plied to the reactor head vent (to main steam) line in this area, c. Reactor Pressure Vessel-Biological Shield Air' Gap Temperature

~

The licensee found that the temperature ' sensing thermocouples were in locations being heated oy the reactor vessel support pins and were not representative of.the air gap temperature outside of reactor vessel insuration packagn. Problems oc-curred in monitoring temperature at this area sin.ce'the design specification for the reactor vessel support lug /to shield wall pins does not allow any insulation of the pin and"its washer assembly. Therefore, the pins become a significant haat source in the limited space available to relocate the teeperature ele-men Reactor vessel thermal growth also creates problems by opening air gaps under the vessel support lugst These gaps allow hot air to flow from behind the vessel insulation into the drywell . The licensee has completed"dotailed improvements for the insulation in this area, such as adding, insulation at the top of the vessel insulation package and sealing the air gaps created under the vessel support lugs to assure ' proper sealing when the reactor vessel expands with operating tem-peratur Locations of the three temperature sensing thermo-couples have been relocated away from thi vessel support pins and lowered into the air gap at the top of the insulation

, package. The team was impressed with the details of these improvements.

.

a b

P

-,,,-,-n. - , - - -

-w-

.

30 High Differential Temperature Reactor Vessel Support' Skirt to Reactor Vessel An average differential temperature of 53 degrees Fahrenheit was recorded between the reactor vessel support skirt and the reactor vessel during Startup Testing. In a letter to the licensee dated August 15, 1986, General Electric indicated that a concern had been raised by the vessel manufacturer because of this differential temperature. The licensee was instructed to limit the vessel to 50 cycles under these conditions. Be-cause of this, the licensee applied insulation to the vessel skirt and expects to lower the differential temperature to an acceptable valu The inspection team found that licensee personnel were addressing these issues in a very detailed manner. A final resolution of these problems requires that the reactor be at its normal operating tem-perature. The plant was in cold shutdown during the team inspectio . Inope'rable Reactor Building to Torus Vacuum Breakers On August 8, 1986 the licensee discovered that the reactor building to primary containment torus vacuum breakers were inoperable because of a design drawing error. The vacuum breakers prevent a negative pressure loading on the containment above 0.25 psid. The vacuum breaker assemblies consist of an air operated butterfly valve which is automatically opened at -0.18 psig sensed by a control system differential pressure instrument and a mechanical check valve which opens at -0.25 psi The design drawing was determined to be in error resulting in the instrument lines being incorrectly routed to the transmitter. High and low pressure lines were reversed from their proper configuration, and this error was not found during prior design, construction, or test phaser. Because of this error, the vacuum breaker butterfly isolation valve would not open with the containment in a vacuum condition. The mechanical check valve would isolate with a contain-ment vacuum, and would be the only containment isolation valve on these lines under accident condition This problem was discovered when the control room operators deter-mined that torus pressure was -0.6 psig on July 4,1986. At that time the licensee identified the specific design error and other related problems with instrument valves and work control procedure The licensee also discovered that one of the differential pressure transmitters was isolated making its associated butterfly valve in-operable and that one of the sensing lines in the torus was taped over for painting. This event is described in Licensee Event Report 50-354/86-056 and was also the subject of a Special NRC Inspection Report 50-354/86-4 a 'q f

.

> The inspection team reviewed this item to determine if it reflected

~

's on .the conduct of the preoperational test program or if it was a

'

precursor to any of the observations made during loss of offsite

, power testing. Although.this problem was clearly one of a design drawing error and inadequate manual isolation valve control, there i was clear evidence that the preoperational test program did not in-

clude full functional testing in which the pressure (or vacuum) at N the containment tap was verified to properly control the vacuum

,. breaker operation. Apparently, the test program was limited to the instrument itself, and did not verify correct high and low pressure . tap connections to the containment. This is an apparent weakness in a program which should verify the design adequacy and t identify error Surveillar,ce tests also failed to detect this i proble >

'

Testing should have provided a detailed checkout of the components during their initial operation, and should have provided a verifi-cation of a correct design, L

i The second issue concerns the adequacy of the control of instrument i isolation valves. During the licensee's-investigation, valves were found shut which isolated one of the differential pressure trans-mitter Further review of the control of instrument valves may be appropriat . Spurious Actuation of the Control Room Emergency Filtration System 3'

Spurious actuations of the control room emergency filtration system g

s occurred on May 5, May 8, July 7, and July 29, 1986. Each event

-

was initiated by the control room ventilation radiation monitor <

These monitors operate in a one of two logic, and were determined

\ to have tripped as a result of voltage drift in the detector's high

. voltage power supplies. The licensee stated that the power supply

,

,'

, ,

problem is caused by high humidity conditions. A design change is being considered to replace the power supplies with a more stable design. These emergency safety feature actuations were addressed i in Licensee Event Reports86-012, -016, -037, and -04 The inspection team found no connection between these events and possible problems in preoperational testing. These events were not considered precursors to problems identified during the loss of offsite power testin s < Control Room "A" Chiller Trips Above 60% Loading On June 12, 1986, the "A" control room chiller was found to trip at loads greater thar. 60 percent. The licensee performed routine

,

corrective maintenance under Work Order No. 86-06-12-004-9. The

>

,

?" \

e

~

hot gas recirculation valve ball float within the chiller was found to be partially flooded. The float was replaced and the chiller was returned to servic This event was not related to inadequate design, preoperational or surveillance testing,~nor was it a precursor to any problems iden-tified during the loss of offsite power tes . Broken Automatic Depressurization Valve Air Line In July 1986, an instrument air line on an automatic depressuriza-tion system valve was broken at its connection to the solenoid valve causing the A solenoid valve to fail. Four other automatic depres-surization system valves remained available to permit the system to meet its minimum performance and reliability requirements. Upon inspection, phy'sical damage to the air fitting was evident indicat-ing.that the valve had been stepped on during primary containment entry. This type of physical damage has also occurred at other BWR sites. due to the location of these valves and the difficulty in performing work in the area. The original Target Rock valve fitting design used 3 joined components for this air connection, but an improved one piece machined fitti.ng was_ used to replace this air connection at 11 out of 14 of the safety-related valves inside con-tainmen The licensee plans to install walkways near these valves to reduce the possibility of physical damage as a future plant betterment item; in the interim, a physical inspection of each valve is required immediately prior to drywell closur This event was not related to inadequate design, preoperational or surveillance testing, nor was it a precursor to any-problems iden-tified riag the loss of offsite power tes . Safety-Related Valve "H" Was Stuck Open During a remote shutdown system startup test, one of fourteen safety relief valves was manually opened and would not reclose. In FSAR Chapter 15, one open safety relief valve represents a potential loss of coolant accident situation for a BWR depending upon plant operat-ing conditions. Normally closed safety relief valve H was found to be stuck open, and was the first such occurrence at Hope Creek although this problem has been found at other BWR plants. Examina-tion indicated that the parallel alignment of redundant air solenoid valves A and B had been altered by physical bending. The amount of bending resulted in the solenoid coil tube being scored intern-ally whenever the solenoid valve was deenergized and re-energize The physical contact involved in this scoring caused the apparent

" stickiness" in operation of the A solenoid for this valve. The licensee has inspected the alignment of the A and B solenoids for

O

.

each valve using a straight-edge, and is now requiring this inspec-tion immediately prior to drywell closure. The bending of the solenoids may be related to the physical access problems described abov This event was not related to inadequate design, preoperational or surveillance testing, nor was it a precursor to any problems iden-tified during the loss of offsite power tes B. Recent Events at Other' Plants Standby Liquid Control System Valve Testing The standby liquid control system provides an alternate method for insertion of negative reactivity for reactor shutdown. The system uses explosive squib valves for injection of sodium pentaborate into the reactor vesse Anoth.er plant had a firing failure of the squib valves during sur-veillance testing. This was caused by a change in plant wiring made during a modification. The wiring change, in combination with use of. explosive squibs supplied with an alternate internal bridge wire arrangement, caused a condition in which the firing circuit con-tinuity monitor indicated an electrical path to the explosive squibs where one did not exist. The required design wiring is one which will monitor and fire squibs supplied with either bridge wire ar-rangement. The particular change only allowed the circuit to fire one configuration, but indicated continuity in both. This event is addressed in NRC Information Notice 86-13 and Supplement No. One potential problem found by the team is that the Hope Creek Operating License Technical Specification 4.1.5.d.1 only requires that "The replacement charge for the explosive valve shall be from the same manufactured batch as the one fired or from another batch which has been certified by having one of that batch successfully fired." The Specification does not identify the standards to which the test firing and certification is to be made. -The concern is that this certification could be made as the result of inadequate bench testing. This situation occurred in the example addressed in Information Notice 86-13. Specific differences between the ac-tual plant conditions and the design assumptions may not be detecte Surveillance procedure OP-ST.BH-002 (Q), "SLC Flow Test" and main-tenance procedure MD-CM.BH-002 (Q), "SLC~ Explosive Valve Removal and Replacement" do not implement the additional recommendations stated in NRC Information Notice 86-13. That notice strongly re-commended that replacement explosive squib units be restricted to those tested in plac o

.

The team reviewed the requirements of Preoperational Test PTP-BH1 and found that the firing circuit continuity was present with either the primary or alternate bridge wire arrangements. The team there-fore conc.luded that a weakness in the standby liquid control system surveillance and maintenance program exists relative to the scope and detail of testing performe . Breaker Preventive Maintenance, Testing, and Coordination The team conducted a review of the licensee's program for preventive maintenance and the extent to which testing is performed on breakers used in the onsite electrical distribution system. Recent reviews at other facilities have identified weaknesses in the conduct of preventive maintenance and testing, especially with'480V molded case breakers found in motor control center The licensee's established program covers 4160V switchgear breakers, 480V unit substation breakers, and 480V molded case breakers. The total. number included in-the program is 1741 breakers, with 682 used in "Q" designated systems. Forty-four breakers located in unit substations and motor control centers have Technical . Specifications specified surveillance testing requirements which are incorporated in the licensee's program. Depending upon the type of breaker and whether there is required Technical Specifications surveillance, the preventive maintenance cycle is 18 months, 36 months, or 54 months. Only the non-Q designated breakers fall into the 54 month cycl An extensive licensee program was determined to exist. The breakers receive appropriate inspections, disassembly, cleaning, lubrication, and testing in accordance with approved written procedures. Fol-lowing preventive maintenance work activities, the licensee's proce-dures and specified instructions on the accompanying work orders identifies the testing to be performed. Time response, relay cali-bration, and overcurrent trip testing activities are performed and documented.~ All testing acceptance criteria are traceable to elec-trical protective device settings that were established by the lic-ensee's breaker coordination design efforts. Selected calculations for trip device selection and settings were reviewed by the team and verified to be properly translated onto the electrical protec-tive device drawings. The licensee uses the system protection group of their transmission and distribution department to perform the protective relaying calibrations for the 4160V breakers. The group's relay. test manual, which provides the instructions used to perform the calibrations, is reviewed by the plant's safety onsite review committe In addition, relay test orders are written that stipulate the work order covering the activity, and reflect the use of elec-trical protective device drawings that are the basis for verifying the proper device trip setting. A minor deficiency was noted that

e

.

involved the system protection group not leaving the completed relay test orders with the site maintenance group. Immediate corrective action was implemented to resolve the team's concer .The team found the licensee's program and its implementation to be effective, and represents a significant licensee strengt . Control of Instrument Root Valves In 1985, an event occurred at an operating facility that rendered the emergency feedwater actuation system inoperable because of in-adequate procedural guidelines and controls associated with instru-ment root valves. In light of this event, the licensee's a.dmini-strative controls, training, and QA oversight for control of in-strument root valves was assessed by the tea Station procedure OP-AP.ZZ-109(Q), Revision 3, " Equipment Operational Control," contains instructions to control valve alignments and verifications. The licensee requires the individual performing only valve alignment or verification to use a valve lineup sheet or a written procedure that specifies which valves.are to be manipulated or verified, and the required position in which the valves are to be left. -Station procedure OP-AP.ZZ-108(Q), Revision 1, " Removal and Return of Equipment to Service," provides a method for removal and returning of specific equipment to' service, and addresses in-strumentation used for reactor scram, containment isolation, engi-neered safety features actuation, system interlocks, and permissive The procedure is implemented in Mode 3 or higher. Another procedure used to control valves, including instrument root valves, is OP-AP.ZZ-103(Q), Revision 2, " Tagging Request and Inquiry System Use, Management, and Audits." The tagging request and inquiry system is a real time data base management system that is used in conjunc-tion with the safety tagging procedure SA-AP.ZZ-105(Q), " Station Safety Tagging Program." All procedures reviewed make reference to applicable INP0 SOERs and NRC I&E Notices to reflect operational assessment feedback concerns for valve mispositioning events in-volving deficiencies in the administrative, procedural, training, and hardware area The administrative controls established in the aforementioned posi-tions appear to provide sufficient depth to preclude the occurrence

~

of a loss of control of instrument root valve The licensee's training program for plant operators in the area of valve fundamentals and operation and administrative directives were reviewed. Training in the above enumerated procedures is conducted and includes discussions on proper valve operation with a review of appropriate incidents covered by operational assessment feedback in the area of valving error Although the specific incident that impacted the team's review at this facility was not incorporated

..

e

in the training program, prior events and feedback assessments envelope the concerns for controls on instrument root valves, and are incorporated into the licensee's training activities and estab-lished administrative control 'The tagging request-and inquiry system alignment output reports for the nuclear boiler and reactor protection systems were compared to the instrument root valves showed on P& ids M-42-1 and M-01- No deficiencies were identified. This review verified that the in-strument root valves are subject to the same level of administrative control as other valves in plant systems. All manipulations of root valves are performed in accordance with approved system operating

. procedures and supported by system lineups contained within the tagging request and inquiry system. However, for the nuclear boiler system, .which is designated as system "BB", it appears that the licensee does not require independent verification. System de-scription "BB" is not listed on a system requiring independent veri-fication activity in station procedure SA-AP.ZZ-002(Q), Revision 6, "S.tation Organization and Operating Practices." The team noted that the instrument root valves in system "BB" provides process variable inputs to other systems (e.g. , reactor protection system, residual heat removal, and core spray) which require independent verification activity to verify the proper position of components that could affect the operating status of the systems. The tagging request and inquiry system alignment output reports indicate that independent verification is not required. ~ Licensee corrective action is warranted to address this concer Quality assurance activities of the licensee were inspected and determined to provide valve verification reviews of instrument root valves as part of the QA surveillance program. Surveillance reports86-674 and 86-803 performed on July 16 and September 15-16, 1986, respectively, contained some instrument root valves. QA audits of instrument root valve alignment have not been conducted. The tag-ging request and inquiry system has continuous, monthly, and quar-terly audit functions performed by a designated component audit co-ordinator of both the tagging system and tagging request and inquiry system data base. The coordinator is also responsible to provide the resolution of problems discovered during audit With the exception of the issue of not providing independent veri-fication of instrument root valves that are part of instrumentation important to safety and the valve mispositioning error discussed-earlier in connection with the vacuum breaker issue, the team.has concluded that an effective program of administrative control and practices now exists that should ensure that instrument root valves are in their proper position to support the safe operation of the plant.

i

!

L l

l

!

. .-

.

37 C. Hope Creek-Generating Station Preoperational and Startup Test Program Preoperational Tests In accordance with 10 CFR 50, Appendix B, Criterion XI and Regula-tory Guide 1.68, the preoperational test program is designed to perform a defense-in-depth approach to assure the adequacy of the design. .The principal focus of the program relates to safety-related functions. Other lesser important functions are demonstrated to a less stringent basi The preoperational test program for Hope Creek was described in Chapter 14 of the FSAR. The program was reviewed and accepted by the NRC through the operating license application process. Docu-mentation of this acceptance is in the Hope Creek SER. The team reviewed selected aspects of the accepted test program described in the FSAR to determine if the-problems encountered during the loss of offsite power testing resulted from an inadequate scope or depth of testin Based on the team's analysis of the hot and cold loss of offsite power test observations, several instances were noted where a more complete preoperational test might have identified these problems earlie In the case of reactor auxiliary cooling chilled water and the instrument air systems, it appeared that more thorough testing would have eliminated a number of test observation How-ever, the team noted that none of these systems was considered safety-related and each system did receive a specific preoperational test. Problems encountered during the loss of offsite power tests resulted from mutual interactions among these systems. The team

. concluded that the testing performed on these systems was generally adequate, and the failures subsequently encountered did not consti-tute significant safety concern However, the team made two significant observations regarding other aspects of the licensee's test program as described belo A preoperational loss of offsite power test was described in FSAR Section 14.2.12.1.47. The test objectives, methods, and acceptance criteria listed in the FSAR for this test and the licensee's answer to FSAR Question 640.11 appeared inconsistent regarding the extent uof this test. The inconsistency involved whether the non-Class IE buses would be within the scope of the tes The objectives and test method descriptions appeared to indicate that a total loss of offsite power would occur. However, acceptance criteria in the FSAR and the licensee's response to the FSAR ques--

tion appeared to limit the scope to only the Class IE buses. Pre-operational test (BB-3) only tested the loss of power to the Class

-- _ - _ _ _ _ _

o

.

1E buse The team noted that had a true total loss of power test had been implemented, some of the recent loss of offsite power test failures might have been identified earlie A second concern involved the extent of preoperational instrument air testing. Regulatory G.ide 1.68.3 specifies, among other things, that an integrated loss of air test be performed. As indicated in FSAR Section 1.8.1.68.3 and 14.2.13.4, the licensee took exception to this item. The licensee committed to do~ individual loss of air tests only on safety-related equipment. The team noted that an integrated loss of air test for all plant components might have identified some of- the problems found during subsequent loss of offsite power testing involving multiple system interactions and dependencie . Startup Tests The team reviewed the scheduling of the September 11, 1986 hot loss of offsite power test to determine if it was performed as described in FSAR Figure 14.2-5. The Figure provides the integrated test schedule that had been reviewed by NRC during the license review proces Condition C (10) to the Hope Creek Operating License (NPF-57) requires that the. licensee perform the program as described in the FSAR, but allows changes to that program provided a safety evaluation is performed pursuant to 10 CFR 50.5 The FSAR specified that the loss of offsite power test be run in Test Condition 1. The test was actually performed in Test Condition 2. The team reviewed the safety evaluation used to justify deferral of the test from Test Condition The safety evaluation was found to be incomplete for two reason First, the evaluation had not been completely reviewed as only one of four of the signature blocks (the initiator's signature) was Records indicated that the safety evaluation had been

~

filled ou accepted by SORC. Second, the safety evaluation did not address whether deferral of the test would result in the plant operating in Test Condition 2 with a reliance on untested equipmen The team noted that similar weaknesses existed in the safety evalu-ation for advancing the schedule for the shutdown from outside the control room test'and approximately ten additional test content or schedule change The team determined that the problems described above partly resulted

~

from the licensee not having an approved station procedure for per-formance of safety evaluations. The lack of this procedure also potentially impacts the adequacy of those evaluations performed to justify authorization of jumpers and lifted leads and other tempor-ary modification e

.

39 Shutdown from Outside the Main Control Room Test

);

'

The team. reviewed startup procedure TE-SU.SV-281(Q), " Shutdown from Outside the. Control Room." The team reviewed this procedure to determine if the root cause for problems identified in this test were similar to the causes for the recent loss of offsite power problems. In the course of its review, the team also examined as-pects of procedures OP-AB.ZZ-130(Q), " Control Room Evacuation," and OP-IO.ZZ-008(Q), " Shutdown from Outside the Control Room."

The team found that there were 12 items identified during the tes In general, the team judged these occurrences as typical of those generally found during such tests. None of these occurrences rep-resented a significant safety concer ' However, there were_ several occurrences worthy of note. One safety relief valve, F013H, stuck ~open during the portion of the test covering the transition from hot shutdown to cold shutdown due to a misalignment and sticking of its air solenoid valve This oc-currence resulted in the cold shutdown portion of the-test being aborted. The response of the licensee's power ascension organiza-tion'to this occurrence was timely and adequate. The final resolu-tion was that the portion of the test from operations to hot shut-down was acceptable, and the demonstration of the transition to cold shutdown would be deferred to a later test condition. This resolu-tion was acceptable in accordance with the FSAR and Regulatory Guide 1.6 The other problems identified involved the operation of reactor core isolation cooling system at low pressures and the fact that the reactor vessel water level indication at the remote shutdown panel was offscale in the high direction for'the-first 40 minutes of the test. The former apparently resulted from a procedure problem, and the latter-problem is being reviewed to verify the acceptability of the instrument's rang The procedure problem involved a caution on RCIC. turbine operation The operator was cautioned against operating of the turbine at less than 2150 RPM due to check valve slamming considerations. However, procedure OP-IO.22-008(Q) required operation of the reactor core isolation cooling system at very low reactor vessel pressures (-80 psia). The reactor core isolation cooling system turbine may not be able to maintain 2150 rpm at these low pressures. The caution may be inappropriate, as written, to allow full use of the reactor core isolation cooling system in control room evacuation scenario Additionally, referring to reactor pressure in absolute instead of gage is inappropriate because the team was told that the pressure indicator at the remote shutdown panel reads in gage pressure.

!

, -, .-

_

e

..

The problem of instrument range was also reviewed by the team. The test was started at low reactor power (approximately 20%) and with the initial reactor vessel water level high in the operating rang .The minimal vessel level shrinkage which occurred after the reactor trip along with the effects of feedwater coastdown flow to the vessel resulted in the reactor vessel level increasing above the indicating range of the remote shutdown panel instrument. Indica-tion from GETARS, a system not readily available during actual con-trol room evacuation events, showed the maximum level was about 68 inches (8 inches above the top of the remote. shutdown panel indica-tor range). The occurrence list item disposition addressed why the high water did not damage the remote shutdown panel instrument, but did not address the operational acceptability of not having a posi-tive on scale level indicatio The licensee committed to evaluate the above concerns. However, the team considered the above problems as not being atypical of those encountered at other facilities in a startup test progra ' FSAR Commitment Tracking As a result of the findings described in Section III of this report re-garding conformance to FSAR commitments related to system design (e.g.,

Regulatory Guide 1.97), the team conducted a review of the licensee's FSAR commitment tracking program. This program, managed by the Public Service Electric and Gas Licensing organization, involved a review by a contractor of the FSAR, the environmental report, the SER, the standard review plan, and other related correspondence to cull out those commit-ments warranting verification. The items selected were forwarded by Licensing to the respective Public Service Electric and Gas Company departments for review and verificatio At a meeting with licensee representatives, the team learned the there was no one single organization responsible to. complete the verification activity. Rather,. each specific department was responsible to the Vice President for completing its assigned actions. Completion was documented in letter from the departments to the Vice Presiden The team reviewed the guidelines and procedures ~ used during the verifi-cation provided as an attachment to an April 21, 1983 letter from the contractor to the licensee. These guidelines clearly defined which com-mitments would be selected for-verification and which would be exclude All . future action items and procedural- commitments would be selected; however, for other types of information, such as that provided in the FSAR, only selected topics would be verified. These topics were to in-clude those statements which addressed the acceptance criteria for FSAR reviews contained in Section II of the various SRP chapter ,__ ._

e

.

The records of this program ~ indicated that a reasonably thorough approach toward commitment identification had been implemented. To determine why the Regulatory Guide 1.97 problems were not identified prior to the loss

.

of offsite power testing, the team focused on the verification of power supplies to the safety relief valve position indicating system, i.e.,

the acoustic monitor The acoustic monitoring system was. verified by the licensee's program in connection with TMI Item II.D.3. Site engineering was tasked to verify this item. Based on available records, the team found that site engineering had credited the statements in a February 24, 1984 letter from Bechtel to the licensee for verification of this item. The letter provided draft responses to NRC questions on the FSAR and forwarded pro-posed revisions to Sections 1.10 and 7.5 of the FSAR. These' revisions included a' statement that the acoustic monitoring' system's power supply conformed to Regulatory Guide 1.97 specifications. The FSAR revisions were included in FSAR Amendment It appeared to the team that verifi-cation by review of .FSAR amendments was a common practice of site engi-neering. ,However, this practice was-weak because it relied ~on revised FSAR statements provided by the architect-engineer rather than on a re-view of onsite as-built drawings or other design information. For this reason, it appeared that the issue of compliance to Regulatory. Guide 1.97 for this particular system was prematurely close E. Regulatory Guide 1.97'Conformance The Hope Creek Generating Station commitment to Regulatory Guide 1.97 was provided in Section 1.8.1.97 and Table 7.5-1 of the FSAR with addf-tional supporting information from FSAR Tables 11.5-1 and 2.3-29. NRC review of the Hope Creek Generating Station commitment was described in SSER 2 where the proposed implementation was acceptable subject to a future determination of upgrading the neutron flux instrumentation.from category 2 to category During the team's inspection, the implementation of Regulatory Guide 1:97

- was reviewe In general, the actual Hope Creek Generating Station im-plementation satisfied the technical requirements of Regulatory Guide 1.97. The following specific issues regarding that implementation were discussed with Public Service Electric and Gas personne . The average power range monitor channels were changed from the reactor protection system power supply to a Class IE uninterruptible power supply to assure their continued availability after a postu-lated loss of offsite power. Intermediate range monitor and source range monitor detector channels are powered from Class IE 24 volt DC battery sources. However, the intermediate range monitor and source range monitor detector drive mechanisms were found to be powered by non-1E interruptible power sources wnich would preclude the reinsertion of the 8 intermediate range monitor and 4 source

_. - _ _ _

O s

range monitor detectors to their effective operating position in the core region in loss of power events. This is, in fact, what actually occurred during the hot loss of offsite power test (H-19). Suppression pool water level is measured over a different range than that specified in Regulatory Guide 1.97. Rather than from the lowest emergency core cooling system suction piping elevation, which is approximately 24 inches above the bottom of the torus, the meas-urement range is from 94 to 274 inches above the torus botto Emergency core cooling system suction would still be available to the pumps after the control room operator loses level indication at 94 inches above torus botto . Indication of the accomplishment of primary containment isolation is from valve " closed" indicator lamps for most of the valves. How-ever, main steam isolation valve position lights are powered from the reactor protection system motor generator sets which.would cause the loss of the main steam isolation valve position ir.dication lights for a loss of offsite power conditio . In addition to the non-Class IE interruptible power supply described in section III.E.4, the main steam line safety relief valve acoustic

,

monitoring detectors have a minimum sensitivity for sensing of leakage flow in the discharge lines; hence, these monitors cannot assure that the safety relief valves are fully close . The stated process ranges of various type D status indication flow instruments, such as feedwater, torus spray, drywell spray, and emergency core cooling system flow instruments, were confirmed to be in accordance with the 0 to 110% of design flow. range specified in Regulatory Guide 1.9 . The stated process ranges of various type D status indication in-struments,' such as condensate storage tank level, standby liquid

. control tank level, and the radwaste tank level were confirmed.to be within the operating levels of the tanks (top of outlet suction pipe to tank overflow outlet.line). These process range values satisfy the Regulatory Guide 1.97 range requirement of the tank bottom to tank to Redundant Reactivity Control System

'

The Hope Creek plant experienced, in the past, a spurious actuation of the standby liquid control system which is automatically initiated by the redundant reactivity control system or may be manually actuated by the control room operator. The spurious actuation was stopped in time to prevent an actual injection into the reactor core. The purpose of the system is to inject a sodium pentaborate solution into the reactor

. . - - .

.- - . - -.

.

core through a high prersure coolant injection piping connection to de-crease the core's reactivity. Final actuation of standby liquid control equipment is accomplished using Bailey solid state logic module The team reviewed the circumstances surrounding the spurious actuation with Public Service Electric and Gas personnel as well as the standby liquid control piping and instrument diagram and control logic diagram Recent system modifications, such as the addition of an Agastat relay and interposing relays to minimize the possibility of redundant reactiv-ity control system self-test pulses from entering the final actuator Bailey logic modules, were reviewed. The licensee has been unable to identify a reasonable cause,for the spurious actuation, and has confirmed that the installed equipment continues to operate in accordance with the design. The team also could find no reasonable cause for the spurious actuation; consequently, the occurrence remains unresolve The team found the General Electric system specification for the redund-ant reactivity control system was ambiguously written with respect to specified. logic time delay values provided in the Hope Creek design specification data sheet. Because of the number of such ambiguities between these two documents, a resolution could not be achieved during the team's inspectio The installed redundant reactivity control system equipment was inspected, and appeared to satisfy system requirements for separation and diversity from the reactor protection syste G. Overview of Quality Assurance During Startup Tests FSAR Section 14.2.2.2.5 describes the role of QA during the preopera-tional phas The licensee implemented this FSAR commitment through procedure QAI 2-11 (Revision 5), "QA Program for Phase I and II of the Startup Program." The role of QA is also described in the site startup manual. QA activities during startup were required only for safety-related (Q) systems and fire protection and radwaste systems. There was no QA involvement for startup activities involving non-safety-related systems such as the turbine chilled water system, and the emergency air syste The QA activities assured that the program was conducted in accordance with FSAR commitment During the preoperational phase, the 23 member QA staff provided coverage for about 95 percent of the safety-related fire protection and radwaste system tests. This involved review of about 2900 procedures, 41,000 QC inspections, and witnessing of 8000 functional test Forty-nine action requests were generated from QC inspection and surveillance, and all startup related action requests have been close The licensee's QA involvement in the reactor auxiliary cooling system preoperational test is an example of the type of QA coverage provided during a safety-related preoperational test activity. Sections 8.2.3, k

-

O

.

8.2.4,'and 8.2.6 of the reactor auxiliary cooling system preoperational test (PTP-ED-1) were identified'as the safety-related portions. QC verified and signed off on the significant steps in these sections. All test exceptions for the reactor' auxiliary cooling system were closed prior to fuel loa QA was heavily involved in the initial programming of field programmable logic arrays in the Bailey module cabinets. However,.QA and the line organization agreed to. terminate this programming coverage as the field programmable logic arrays were individually tested. prior to declaring the systems operational. This action is consistent with the licensee's

~

commitment regarding QA' coverage for the startup test progra QA cor. ducted 7 audits in the startup area since June 198 Corrective action requests were initiated to resolve identified program concern These concerns were closed in a timely fashion and the audits did not identify any significant safety or hardware concern QA involvement in the power ascension test program is provided by proce-dure QAP-5-1.1 (Revision 2), " Surveillance of Phase III Startup Test."

Surveillance is the primary means to provide QA coverage. The program requires mandatory surveillance of 25 percent of all safety-related power ascension tests. Additional random QA surveillances are also performed during the power ascension program. No significant action requests have been generated from this QA surveillanc The QA coverage for the startup and power ascension test program is con-sistent with the licensee commitments. QA surveillance and inspection were also observed to be effective during the preoperational test by various NRC inspectors. QA's role in resolving the hot and cold loss of offsite power test observations was effective and visible. The QA personnel were. knowledgeable in the activities they were responsible for witnessin V. Bailey Logic Modules Implementation of balance of plant control logic functions at Hepe Creek for both safety-related and non-safety-related electrical'and instrumentation circuits has been accomplished using three types of Bailey solid-state com-ponent printed circuit board logic modules. Two of these module types, namely the main control room pushbutton module and the reed relay output driver module, have not been subject to the number or types of failures experienced by the third type, the Bailey 862 logic module. The Bailey 862 solid-state logic module performs a variety of functions that include logic gates, timers, interlocks, and signal conditioning. The programming flexibility provided by the field programmable logic array chip mounted on the Bailey 862 logic module makes it easily adaptable to meet a variety of configuration . - . -

--

-

+

Solid-state modules were first used in nuclear generating plant safety systems in the early 1960's (i.e., Big Rock Point), but significant progress was not made toward in replacing relays in some safety system designs until the mid-1970's. Since the Hope Creek plant is the only nuclear plant using the Bailey logic module in safety-related applications, Public Service Electric and Gas Company.has implemented specific design modifications and established ~a re-liability assessmentiprogram to improve the performance of the Bailey logic module. The Hope Creek plant uses 1138 of these modules in safety-related applications with 573 modules performing active functions on approximately 250 field components. Another 1120 modules are used in non-safety-related application Bailey Logic Module Failure Rate At the present time, the Bailey. logic module appears to have a signifi-cant failure rate. During the most recent ten month period, more than 80 logic module failures have been recorded out of a total population of 2258 modules which represents an approximately 5 percent annual fail-ure rate. This failure rate appears higher than desired _for engineered safety feature system component In addition to a number of non-repeatable intermittent type failures, the failure rate for input and output buffers on the Bailey logic module has been significant. Module troubleshooting after the loss of offsite power tests identified 7 failures in input or output buffers. During the hot loss of offsite power test, 2 buffers failed that.are associated with two breakers that supply normal power to the "C" emergency bus pre-venting the "C" diesel generator breaker from closing. Concurrent fail-ures of this nature raise significant safety concerns, especially when these failures are revealed only when called upon. Two different buffers of the same type failing on different cards could have prevented 2 diesel generator breakers from closing. Other factors that could be impacting the observed failure rate include its susceptibility to human error due to removal and reinstallation of the field programmable logic array chip and input signal staple jumpers, the abse~nce of a full in-situ confirma-tion test that assures the operability of all safety functions imple-mented by each logic module, and the number of electric circuit friction connections at the logic card edge connector, field programmable logic array chip socket, and input signal staple jumpers. Specific Bailey logic module problems observed during the hot and cold loss of offsite power tests are discussed below: Safety Auxiliary Cooling System Pump B Did Not Start (H-10)

The Class IE safety auxiliary cooling system pump B did not start from the diesel generator sequencer, and the root cause of this failure has not been determined. During the module troubleshooting process, the Bailey logic card buffer for the process start inhibit

a

.

signal was found to have failed; however, this failure could not have prevented the pump from startin During the cold loss of offsite power test, this module functioned properl . Emergency Diesel Generator C Output Breaker Failed to Automatically Close (H-24)

The emergency diesel generator C output breaker did not clos The output buffer on the Bailey logic card for each of the two alternate incoming breakers failed such that there was no signal provided to permit closing of the diesel generator C breaker. Either one of these failures would prevent this breaker from performing its in-tended safety function. During the corrective action taken for this problem, an unrelated drawing error was found on one of the load breakers on the 4160V C vital bus; however, the field installation was in conformance with the design change requirement . D'rywell Recirculation Fan Failed to Start (C-1)-

Drywell fan 1G1-VH212 failed to start due to a misaligned Bailey logic card connector pin in panel 1AC65 . Process Start Inhibit Signal Did Not Inhibit Reactor Auxiliary Cooling System Pump Manual Start (C-2)

After completion of the cold loss of offsite power test, reactor auxiliary cooling system pump BF209 was manually started. A process start inhibit signal should have prevented this action, but did not

'because of incorrectly positioned staple jumper-1 on a Bailey logic car Jumper 1 was positioned for a 120VAC signal input instead of 24VDC, preventing logic module from recognizing the presence of an inhibit signal. Staple jumper 5 on the same module was also mis-positioned. It would have caused failure of an indicator light to tell the operators that a bus power failure had occurre . Control Room Air Fan Did Not-Start (C-7)

Control room return air fan.BV-415 did not start after the cold loss of offsite power tes This was due to the failure of a 20 second time delay module on Bailey card 4-10-4 which prevented the fan from startin . 7.2KV Supply Breaker Timer Module Failure (C-13)

The alternate 7.2KV supply incoraing breaker closed when offsite power was restored. This action occurred automatically without operator interventio Upon a trip at the normal supply breaker, the operator is required to depress the' flashing trip ligh This resets the memory function within the Bailey logic module to prevent automatic closure of the alternate breaker when offsite power is

C

.

'

restored. Since the timer module failed in the flasher circuit for the normal supply breaker trip. light, the trip indicating push but-ton did not flash and the operator did not reset the memory of the Bailey logic car In the absence of this reset, the alternate supply breaker functioned in accordance with the design by auto-matically closing onto the bu . Automatic Breaker Close Input Buffer Failure (C-14)

The normal incoming circuit breaker 52-10101 to bus 10A101 closed when the automatic close-block was de-selected. After the loss of offsite power test was initiated, circuit breaker 10101 tripped on undervoltage. The trip light fl. ashed as this was an uncommanded (i.e., not initiated by the operator) trip. The operator then acknowledged the trip and flaching of the trip light stopped. The Bailey logic module input buffer transmitting the undervoltage sig-nal failed, which caused the breaker to automatically close when automatic close block was de-selected coincident with feeder voltage being restored. The malfunction could not be repeated on September 19. This particular failure was finally located and corrected when the same problem repeated during the second cold loss of power test on 0ctober 2, 198 ~

To further evaluate the extent of possible Bailey logic module defects, a sample of 133 modules was chosen by the licensee for reinspectio Each of the selected. modules had at one time two concurrent design changes pending for Hope Creek onsite implementation. One concern was the possibility of an out-of-sequence implementation of individual' design changes that could lead to incorrect programming of the field program-

~

mable logic array chip. From this sample, the following problems were

, identified: Defective Bailey logic module memory A output for chilled water AK40 . Bailey logic module input buffer failure for safety auxiliary cool-ing system "C". Circulating water pump Bailey logic module output buffer number 4 failur . Bailey logic module input buffer number 7 failure on filtered re-circulation ventilation system "D". Bailey logic module program errors in vital bus incoming breaker and 480V load center breaker proble .

- -- - - - -

o

,

..

48 Safety auxiliary cooling system B pump drawing was in error even though the field installation was correc ~

In addition, the team inspected other aspects o'f the design installation and use.of Bailey logic module as follow Interdependency Between Automatic and Manual-Protective Functions As stated in IEEE Standard 279-1971 (10 CFR 50.55a(h)), the amount of safety system equipment _ that is common to the initiation of automatic and manual protective actions should be minimized to limit the probabil-ity of their concurrent los Industry practice in implementing this criterion has been to avoid having common equipment for automatic and manual protective actions within the safety system circuits. This prac-tice also recognized that such dependencies in output electric circuits to the actuated equipment, such as pumps and valves, could not be en-tirely avoide Nevertheless, with careful and prudent circuit design, the amount of interdependency can be kept to a minimu In the Hop'e Creek _ engineered safety feature circuits using Bailey logic modules, the amount of equipment common to both automatic and manual

. initiation circuits for various protective actions is more than tha typically provided by the design practices in relay logic circuits used at other nuclear plants. This increased use of common equipment raises the probability that failure of a Bailey logic module will disable both automatic and manual circuits for a given piece of actuated equipmen As a result'of this concern, the team recommended that the second cold loss of offsite power test conducted on October 2, 1986 include exercis-ing of a representative sample of Bailey logic cards that have both manual and automatic functions. The cold loss of power test performed on October 2, 1986 included such exercises with no significant failures as described in Section V C. In-Situ Testing Capability A dominant factor in the achievement of high reliability for safety sys-tem channels and logic trains is the capability to perform on-line peri-odic tests to confirm proper circuit operation while the individual com-ponents remain installed in their normal system configuration. The Hope Creek engineered safety feature systems have only a limited capability for in-situ testing of channels and logic train For example, approxi-mately 50 cercent of the Bailey logic modules can'only be fully tested at refueling outages since the system design prevents an effective in-situ test during power operation. Even during these refueling outage tests, only a limited number of logic combinations are examine n l

.

49 Field Programmable Logic Array Programming Errors A manual method is used to enter and store a. specific program on the field programmable logic array chip used in the Bailey logic modul Design drawing data is entered into the programmable chip by typing at a programmer-computer keyboard and observing the storea program results on a console cathode ray tube.or on a hardcopy printer output recor The program is usually only visually checked on the screen against the drawing. This checking process does not actually verify its capability to perform the required functio Hard copies of stored program results have not been kept until recently. In-situ surveillance checks performed at the component or system level verify that the correct signals are generated under the specified conditions;.however, this process does not verify that incorrect. signals are not present. Several of the non recur-ring and unexplained failures in Bailey logic card actuated circuits raise a concern in this regard. The team was concerned about the degree of uncertainty in correct programming of the chip since Public Service Electric and Gas Company had not been documenting the accomplishment of an independent check of the stored program after system turnover from the Start-Up organization. Another team concern involved the possibility

'of unrevealed' static sneak circuits in the programmed logic chip, parti-cularly since several chip programming errors had been identified. On September 15, 1986, a specific requirement for independent verification of future field programmable logic array chip programming and placement of input signal staple jumpers was implemented by a revision of procedure IC-GP.ZZ-031(Q). Staple Jumper Placement Errors On the Bailey logic module, a selection of input signal circuits is made to match 120VAC, 125VDC, or 24VDC voltage levels provided by field input This selection is accomplished by manually inserting a " staple jumper" into one of.three possible location Each module has up to 16 staple jumpers which must be correctly placed on the module. Incorrect place-ment can result in failure of input buffers to respond due to lack of input sensitivity, or damage could occur if buffer input sensitivity is too high. During initial system design, such flexibility is a desirable characteristic; however, the process used for module bench testing in-troduces a susceptibility for human error during reinstallation of the staple jumpers. The current bench test procedure requires removal and reconfiguring the jumpers prior to testing, and again after tes. ting to assure each input has the proper voltage sensitivity. Several staple jumper placement errors (=4) have been identified among the Bailey logic module failures to dat Bench Test Capability After removal of a Bailey logic module from the installed system configu-ration, various bench tests may be accomplished using a module test set and a programmer device. However, the extent of bench testing performed l

i

O

.

O

has been limited. Aspects of this bench test may actually detract from the accomplishment of system reliability objectives. The bench test of a Bailey logic module requires that the field programmable logic array chip and input signal staple-jumpers be removed from the printed circuit board. After the field programmable logic array chip has been separately tested in the programmer and reinstalled into its socket on the printed circuit board, and the staple jumpers are rearranged, no test set cur-rently available onsite permits a thorough retest of all functions im-plemented by the Bailey logic module as a completed assembl Radio Frequency and Electromagnetic Interference Capability In the past, the installed configuration of Bailey logic module at Hope Creek has demonstrated a susceptibility to radio frequency and electro-magnetic interference. Corrective fixes were implemented by adding

" sister boards" within each Bailey 862 module; however, this change pro-duced a number of problem Time delay adjustments and expanded admini-strative controls over the use of portable radio transmitters and arc welding equipment in the vicinity of the Bailey equipment appears to have resolved most, if not all, of the interference problem STAG Field Programmable Logic Array Programmer Calibration Programming of a Bailey 862 field programmable logic array chip involves the removal of fusible links on the chip (i.e., " burn-in") in a format that corresponds to the required formation of logic gates, inputs, and outputs. A programmer-computer manufactured by STAG is used.to accomp-lish this activity. The team held discussions with the manufacturer of the logic chip and the programmer and the licensee regarding the need to periodically calibrate the programmer-computer to assure a good qual-ity " burn-in". The programmer-computer is currently not calibrate The procurement of a calibrating device is currently being considered by the license Failure Analysis To date, the Public Service Electric and Gas reliability and failure analysis program for the Bailey logic module has not performed or ob-tained a detailed analysis of each failed modul Further, their failure analysis program has not included components associated with the logic module such as cable connectors, fiber optic components, and plugs and receptacles in the cabinet VI. Overview of the Cold Loss of Offsite Power Test on October 2,1986 To assist in assessing the readiness of the plant for a subsequent loss of offsite power test, the licensee decided to run a second cold loss of offsite power tes In accordance with Region I Confirmatory Action Letter 86-12, issued September 24, 1986, the concurrence of the Augmented Inspection Team leader was necessary prior to the licensee running this tes ._ .- ._ .-

,

~d s

,N ,

e .

'

The Augmented Inspection Team reviewed the' licensee's proposed test procedure :

and its. plans for observing the test and documenting the results. The team-found the licensee's plans and p'rocedures satisfactory as written, but re-

-quested that additional tests be conducted during the loss of offsite power test to further exercise manually controlled functions executed through the Bailey logic module system. These additional tests typically involved manual operation of equipment to show that these operations were either achievable or inhibited, based on the system design.' The licensee incorporated.these i

. additional test The licensee performed the cold test on October'2, 1986. -This test.was wit-4 nessed by the team. During this test, about 55 components were manually ex-ercised in.the performance of supplemental Bailey-related testin The results of the test were reviewed during a post-test critique on October 2, 1986, attended by the team. The licensee concluded that the acceptance

- criteria for the test were met. 'However, 16 additional observations were mad , ~0ne.of these observations,-related to a breaker control function was a repeat i of a problem found during the last cold test whose cause had been_ initially

~

!

'

listed.as unknown (C-14). An input buffer had faile It'was subsequently determined to be related to a Bailey module failure. Additionally,'a failure

j of a control room ventilation fan was.a repeat of a cold test finding (C-7).

The C-7 finding was attributed-to a Bailey time delay module failure, the

-

.

October 2 finding was attributed to a. process-related phenomenon.

'

.The remaining 14 observations were deemed by the team an'd the licensee to be '

, minor.in nature. The supplemental Bailey tests- showed no a^dditional failures.

- -

'

VII. Exit Meeting

The Augmented Inspection Team met with the licensee on October 3, 1986 to present its preliminary finding In attendance at this meeting were senior L licensee management representatives, including Mr. C. A. McNeill, and senior

NRC Region I managers. The licensee presented no major disagreements with

! the factual findings of the team.

!

,

1-i x

5

e i

i

- . . , _ _..-..-_;_,,-._,_._.,_-__--_,

IR 05000354/1986050

Text

REGION I

Navigation menu

Search