ML20151A581

Reliability Summary Rept for Bailey 862 Solid State Logic Module

ML20151A581
Person / Time
Site:	Hope Creek
Issue date:	03/31/1988
From:	Massaro M Public Service Enterprise Group
To:
Shared Package
ML19302D405	List: ML20151A578 ML20151A581 ML20151A660
References
NUDOCS 8804070091
Download: ML20151A581 (17)

Text

_ ________

M AR C H 31,198 8 PSRG The Energy People l

l RELIABILITY SUMM ARY REPORT FOR THE B AILEY 862 SOLID STATE LOGIC MODULE l l

,n .,--,--.,.,,,_.,~,... - --,4., . ., e.

_ , r7...,-,,..,.... .

,q3 17..

(( k

. :ff~f h'h <

~

'l;' GIl e o-

+, u.

• p,4 .

..,.m

i$4 2 Mh if.f. T((,f,,'[liha ,x > ' (, g y[I ~!f%j #

v>llk 2' 's ' 1 '-

' .s j, '!!<

, ' , 1ok a.[g .; , ,-

gR 4070091 8a0404 ADOCK 0500 4 {i.....G. -

-;;, g +

i P

ATTACEMENT 1

.j .

1 RELIABILITY

SUMMARY

REPORT FOR TIE BAILEY )

862 SOLID STATE LOGIC MODULE I a

1 l

s

Prepared by : Public Service Electric and Gas Co.  :

j Nuclear Department j Engineering and Plant letterment 1

Original Report Issued : March 31, 1988 i

k J

TABLE OF CONTENTS SECl log EAGI TITLE PAGE TABLE OF CONTENTS 1 1.0 SCOPE 2 2.0 PURPOSE 2 3.0 DIOCUSSION 2 3.1 SSLN FAILURE ANALYSIS 2 3.2 ACCELERATED AGING PROGRAM 3 3.2.1 BUTTER INPUT "PICKUP" 3 3.2.2 BUTTER INPUT "DROPOUT" 5 3.2.3 LOGIC LEVEL OUTPUT VOLTAGE $

3.2.4 RELATIVE HUMIDITY TESTING 5 3.2.5 NONCONTORMING SPECIMENS 7 3.2.6 CAPACITOR TESTING 7 3.2.7 TAILURE RATE 8 3.2.8 COMPONENT TAILURES 9 3.3 IN-ROUSE DATA ASSESSMENT PROGRAM 9 3.4 OTEER 862 SSLS USERS 10 3.5 IN-SITU TESTING FEASIBILITY STUDY 10 3.6 AUTOMATED SSLM BENCE TESTER 12 4.0 SUNNARY/ CONCLUSIONS 12 5.0 SIGNATURES 13 Page 1 m

1.0 SCOPE The scope of this suasary report includes those programs which were. performed through the first cycle of Hope Creek operation and are intended to demonstrate the reliability of the Bailey 862 Solid State Logic Modules (SSLM) used at Hope Creek Generating Station. These programs include an Accelerated Aging progran performed by Wyle Laboratories, a $3LM failure analysis performed by Bailey controls Co., an In-Situ Test,ng $ Teasibility Study performed by MPR Associates, and an in-house failure documentation and review program performed by PSE&G.

2.0 PURPOSE The purpose of this report is to provide discussions of the siCnificant aspects of the Bailey 862 Solid State Logic Module Reliability analysis programs, includirg resulting conclusions, justifications, and reasoning for l the actior:s affecting the course of these programs.

l 3.0 DISCUSSION Public Service Electric and Gas Co. agreed to undertake several programs intended ta expeditiously determine the reliability of the Bailey 862 Solid State Logie Module, and report the results of those programs to the NRC prior to plant restart following the first refueling outage. The following discussion is intended to provide additional clarification and information which is not readily evident from the final reports of the individual programs.

3.1 SSLN FAILURE ANALYSIS PSELG contracted Bailey Centrols Co. to perform a failure analysis on the $$LM's which malfunctioned while in-service at Hope Creek. This was accomplished by gathering all the nonconforming logic modules which were available in November of 1986 (34 SSLM's), and returning thes to Bailey Controls Co. for analysis. The analysis program was broken down into a three phare program in which the module failure mode and defective components Page 2 s

vere identified in phases 1 and 2. Phase 3 included the failure mechanism analysis on the dwiective components.

The results.of phases 1 and 2 are included in Bailey Report QR-5106-E93-75, which provides a breakdown of the defective components for dach module.

The high incidence of 4N36 Opto Isolator failures identified in the report prompted PSE&G to request further information as to the suspected cause of failures and the failure modes of the components.

Bailey representatives indicated that all of the defective 4N36 Opto Iso 14 tors failed to meet Current Gain specifications, which caused the input buffer threshold levels to be out of their specified ranges.

Bailey representatives were unsure of the root cause of the Opto isolator failures, although they speculated that the devices may not have been properly screened upon receipt.

The faulty opto Isolators were subsequently analyzed in phase 3, where no physical damage to the components could be found to account for tolerance deviations.

The 4N36 Opto Isolator, ULN2001A Buffer output driver, and 4050 Logic I/O gates experienced 84 % of the total component failures.

The Buffer output driver and Logic I/0 gates are field interfacing devices which could be directly overstressed as a result of testing or troubleshooting errors. The incidence of failures and failure mechanisms of these components supports the conclusion that many of the failures were externally induced 3.2 ACCELERATED AGING PROGRAN PSE&G contracted Vyle Laboratories to perform an 1

Accelerated Aging Reliability Analysis Program, where 26 SSLM's were tested after being exposed to numerous stresses intended to simulate 2. ' and 10 year module lives.

3.2.1 BUPTER INPUT PICKUP VOLTAGE Initial Baseline functional testing at Wyle laboratories indicated that several module inputicutput voltage levels did not fall within the rang specified by the manufacturer. These deviations are focumented in Notice of Anosoly NO. 2, and are in some cases similiar to the Page 3

_________ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _ ~

inconsistencies previously experienced in the Bailey Failure Analysis program. It should be noted that Notice of Anomoly NO.2 identifies several different types of inconsistencies which will be discussed in further detail below. The input buffer pickup and dropout voltage deviations would result from the Opto Isolator out of-tolerance coadition previously discussed.

As discussed in the disposition to Notice of Anomoly No.2, PSE&G directed Wyle to test the out-of-specification Input Buffers at the ele"ated voltage levels (125 VDC, 118 VAC). The subsequent data indicates that the pickup voltage inaccuracy associated with the Opto Isolator tolerance is greatly reduced or eliminated in the high voltage circuits. It was decided to leave these modules in the test population to determine if the Opto Isolator characteristics drifted with age. The subject modules are repeatedly discussed throughout the test program in Notice of Anomoly No.s 2,3,4,6,7,8,9, 10,12,14,17 and 18.

Review of the final test data from the Accelerated Aging Program indicates that while the Input Buffer pickup voltage levels did increase with time, the '

magnitude of the drift alone, was not enough to account for the deviations which were documented in Baseline testing.

The data from both the Wyle and Bailey programs indicates that the high voltage Input Buffer pickup levels did not continuously drift, but tended to completely fail once the specification value had been marginally exceeded (approximately 3 VDC). Subsequent data from both the Bailey and Wyle programs indicates that none of the anomolous Input Buffers deviated when set to 118 VAC, and 3 deviated when set to 125 VDC.

The 125 VDC deviations were either less than 3 VDC or the Input Buffer was found completed failed.

The Class lE 862 system design at Hope Creek utilizes regulated redundant auctioneered 24 VDC supplies fed from battery-backed regulated output inverters. This design ensures that the minor variations in the low voltage Input Buffer pickup values will not have an adverse affect on the system operation. An Input Buffer pickup voltage which exceeds the 24 VDC supply would be identified in surveillances as any other failure.

Page 4

- ~. - - _ _ -

j

3.2.2 BUFFER INPUT DROPOUT VOLTAGE As discussed in !!otice of Anomoly No. 2, four Input Buffers werg found to have out-of-specification dropout voltages. The disposition to that same anomoly explains that the dropout values were relatively close to the required values, and the minor deviations would not have an adverse impact on circuit operation. The Input Buffer dropout voltage deviations are documented throughout the test program in Notice of Anomoly No.s 2, 5, and 18. Bailey controls determined that the dropout voltage deviations were also caused by the Opto isolator tolerances.

3.2.3 LOGIC LEVEL OUTPUT VOLTAGE As discussed in Notice of Anomoly No. 2, many of the logic and memory output "0N" voltages were found to be out-of-specification during Baseline testing.

A review of the test parameters indicated that the logic output load which was being used was overly conservative when compared to the actual Hope Creek configuration. ,

A plant system review was performed to identify the worst case Logic Output Load. As a result of that review, the Logic Output Test Load was changed to 4892 ohms and all logic and memory outputs were subsequently found to be in-tolerance. The worst case design load was identified to be the circuit configuration which drives two parallel Delay module inputs from a single logic ,

output.

Notice of Anomoly No. 12 documents out-of-tolerance memory outputs on module serial number 0138. The 0.02 volt deviations (4.4 v - 4.38 v) are considered to be insignificant as the logic input threshold level of the receiving module is specified to change state at less than 3.5 volts.

l 3.2.4 RELATIVE HUMIDITY TESTING 1

Functional testing of the SSLM high relative humidity limit was complicated by several factors including misinterpretation of the manufacturers specifications, testing inadequacies, and physical difficulties in controlling relative humidity at 90 percent while ensuring the absence of condensation.

l Page 5 J

l

[

The temperature and humidity extremes specified for l testing of the SSLM's (140 degrees F and 90% RH) were taken from the manufacturers Product Instruction for the device and incorporated in the PSE&G Test Specification.

i The temperature and humidity functional tests were combined in the test procedure developed by Wyle and reviewed by PSE&G. During performance of the humidity portion of the Baseline Functional Testing several anomolies occurred which triggered dialogue between Wyle and PSE&G. There was concern that the anomolies may have been a result of an improper test sequence, and therefore, PSE&G directed Wyle to re-attempt the test after allowing the modules to dry-out. As discussed in Notice of Anomoly No. 3, the second attempt was aborted after 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and 44 minutes due to a similiar set of anomolies.

l Representatives from Bailey Controls Co. were contacted and informed of the inconsistencies encountered in the test program. The Bailey representatives explained that the upper limits of temperature and relative humidity I are not intended to occur simultaneously, and that the original qualification of the SSLM was performed using the nonconcurrent extremes. As a result of Bailey's clarification, PSE&G directed Wyle to modify the test procedure as discussed in the disposition to Notice of Anomoly No. 3.

Notice of Anomoly No. 17 explains that the 10 year humidity and temperature testing was, once again, incorrectly performed at the concurrent maximum limits of temperature and relative humidity. This testing deviation occurred as a result of a change in Wyle test personnel and an oversight in the procedure revision.

The Relative Humidity test procedure was found to be flawed during the Post-2 Year Humidity Operational Test.

As documented in the disposition to Notice of Anomoly l No. 6, opening the humidity chamber door during the humidity test caused moisture to accumulate on the test rack and specimens. The accumulation of moisture was determined to be the root cause of many of the previous l anomolies experienced during Baseline and 2 year humidity testing.

The Test Procedere was modified to eliminate the need to repeatedly open the chamber door during the humidity test.

l Page 6

_)

Notice of Anomoly No. 17 documents several unexplained events which occurred during the course of the 10 year humidity testing. It is our conclusion that these events were once again caused by condensation. The fact that all modules were affected by voltage dips indicates that the 9 VDC supply into the test fixture experienced transients, probably as a result of water tracking similiar to that which is documented in the Disposition to Notice Of Anomoly No. 7, for module S/N 0511.

Subsequent sections of this report explain that PSE&G is undertaking a modification which will limit the relative humidity of the SSLS environment to a maximum of 60%.

This change makes the test program humidity testing extremely conservative.

3.2.5 NONCONFORMING SPECIMENS Three of the test specimens sent to Wyle were found to be missing the modification to the solder pads of the memory toggle switches and the associated conformal l

coating. These modules are identified in the disposition to Notice of Anomoly No. 3, as serial numbers 0804, 0373, and 09L9. These modules were found to be part of a small population which had been sent to Bailey Controls Co. for repair and modification, but were returned repaired without the modification. The total population of suspect modulee has been identified and will be

! removed from the Hope Creek system as necessary. -

Following the changes to the testing procedure, the l

three unmodified modules operated successfully until

! serial number 0373 experienced a component failure in 10-Year logic cycling which is documented in Notice of

( Anomoly No. 15.

l

! 3.2.6 CAPACITOR TESTING The 2 microfarad input buffer capacitors were tested I at each aging interval of the test program. Two of the capacitors were found to be out of tolerance from a total population of 416 capacitors (2 per buffer X 8 buffers per module X 26 test modules). Both of the out-of-tolerance capacitors failed such that they would not have impeded the operation of their associated input buffers. The out of tolerance capacitors are documented in Notice of Anomoly No.s 1, 7, 11 and 18.

Page 7 l

j

l 3.2.7 FAILURE RATE There were five module failures which occurred in the Wyle test program excluding the failure of serial number 0511 which is documented in Notice of Anomoly No. 7.

This module is concluded to have failed as a result of condensation.

Table f i provides a breakdown of the module failures along with the failures rates for each period, in failures per million hours of service. The failure rate calculations assumed that the average module life was 18.5 months at baseline testing. This assumption is based upon the fact that approximately 20 of the 26 modules in the test population were taken directly from operation at Hope Creek with 2 years of service.

The failure rate values are calculated with 2 failures at t= 18.5 months, 3 failures at t= (18.5 + 24) months, l

and 5 failures at t= (18.5 + 60) months. The final failure rate value is based upon 5 failures over a period of t=(18.5 + 120) months.

i TABLE i1 FAILURE / I I FAULTY l N.O.A.lFAILURE l SERIAL NO I OCCURRED l COMPONENT I NO. I RATE

__________..+.______________+_______________+_______+_______

NO. 1 1 IN BASELINE I NOT I I2 1 5.7 l (S/N 0799) 1 TESTING l DETERMINED l 1 i

.._____ ....+_______________+_______________+.______+______.

NO. 2 I IN TWO YEAR I U27 (4050) I i4 1 5.7 (S/N 0390) i TESTING l LOGIC INPUT I l

..._________+-.______.._____+__...._________+_____..+_____ .

No. 3 I IN POST 5 YEARI CR18 1 1 10 1 3.8 l (S/N 0174) l TESTING l OUTPUT LED i i

__ ...______+_______________+....__________ + ...___+_....__

l NO. 4 1 IN TEN YEAR I U3 1 0 15 1 3.4 l (S/N 0373) i TESTING l OPTO ISOLATOR I l

____.....___+...__.__.______+_-___.______.._+_______+_______

No. 5 i IN POST 10 1 CR13 I i 18 I 3.4 l (S/N 1817) ! YEAR TESTING l OUTPUT LED 1 l

..________..+-______________+__.___...______+_____._+______.

l l FIVE FAILURES TOTAL l 1.9

..___________________ ......________________________+....___

Page 8 l

l

[ ._

l The failure rate calculated from in-service failures at Hope Creek over the last 12 months period is equal to 1.8 failures per million hours of service (see section 3.3 for disgussion of Hope Creek in-service failure rate).

It can be shown that the SSLM failure rate experienced through the Accelerated Aging Program was slightly greater than that experienced at Hope Creek. This would be expected, as all of the testing stresses that were applied were conservatively selected. It should be noted that the test program failure rate does not include the failure which was determined to have been directly caused by over-test (S/N 0511).

l 3.2.8 FAILED COMPONENTS

[

The components which failed in the Accelerated Aging i

Program are outlined in Table f 1. While the component failures do not differ significantly from those identified in the Bailey Failure Analysis, the data does l not support any strong conclusions. The open etch failure is not consistent with previous failures experienced, supporting the conclusion that the failure was caused by condensation.

i 3.3 IN-HOUSE DATA ASSESSMENT Pit 0 GRAM l

PSE&G performed an."in-house" data assessment program which included tracking in-service module failures on a monthly bacis.

l The in-service SSLM failures vs. time are graphed in

! Attachment i 1 for the period of March 1986 through l February 1988.

! As a result of the unexpected increase in failures which occurred in July 1987, PSEEG re-examined pertinent parameters in an attempt to identify the cause of the increase. The re-examination of these parameters, together with several additional months of data has led to the conclusion that the increase in failures l

experienced through the summer months is a result of higher average Relative Humidity in the Lower Equipment Control Room (LECR), which houses the 862 system.

While Relative Humidity is maintained within the design basis of the room, and within the qualification limits Page 9 l

l d

of the SSLM, the average Relative Humidity was found to

( increase by approximately 30% (35% to 65%) from Vinter to Summer months.

The dat" indicates that a lag time exists between changes in LECR Relative Humidity and changes in SSLM failure rates. To eliminate the unnecessary stresses associated with the high humidity, PSE&G has begun design of a modification which will maintain the LECR RH between 20% and 60%. Portions of the change are scheduled to be installation complete in June 1988.

3.4 OTHER 862 SSLS USERS PSE&G contracted Bailey Controls Co. to monitor other users of the 862 Solid State Logic System and provide a report of the failure rates experienced at their facilities. The Bailey Controls Co. sunmary reports are provided as part of this submittal.

The information was compiled by submitting questionaires to individuals representing the following organizations which utilize the 862 SSLS.

1. Utah Power and Light Co. - misc. power plant control

2. Colorado UTE - misc. power plant control

3. Associated Electric Power - Burner Safety System

4. Inland Steel Co. - Burner Safety System The most recent report indicates that the other users experienced 24 SSLM failures over 41,749 module-months of operation. This equates to a failure rate of 0.8 failures per million hours of service.

While this program may provide information as to the potential reliability of the system, it is our opinion that the data is less reliable than that obtained from the in-house program. The additional inaccuracy should be considered when comparing the failure rates from the two programs.

3.5 IN-SITU TESTING FEASIBILITY STUDY PSE&G contracted MPR Associates, Inc. to perform a study of the feasibility for in-situ testing of the Bailey 862 system used at Hope Creek. The report generated as a result of that study is included as part of this submittal.

Page 10

l PSE&G made a conscious effort to provide as little input as possible to this effort, so as to not bias MPR's decisions in determining the most cost effective method of modifying the system to incorporate the in-situ testing feature.

The logic module costs discussed in section 3.4.1 of the MPR report are based upon a Budget Estimate provided by Bailey Controls Co. at MPR's request. It is PSE&G's position that the final cost of redesigning the logic module, providing 12 prototype and 600 production run modules, and qualifying the new device, would actually cost significantly more than the S 1,065,000.00, estimated by Bailey Controls Co, since the cost of the replacement modules alone exceeds tne MPR estimate.

l Section 3.4 of the MPR report also provides time estimates for incorporation of the new device. It is our opinion that these estimates are extremely optimistic and can not consider delays associated with material availability, design conflicts resolution, resolution of qualification testing anosolies, device installation, and plant system retest.

Although MPR concludes that in-situ testing is feasible, the method proposed would require a redesign of the SSLM, effectively resulting in a new device. This method does not incorporate a self test feature on the system but redesigns the module to permit it to be partially tested without being removed from the system. This method of in-situ testing would require extensive system impact coordination when performed with the unit on line as it creates the potential of inadvertent l

signals capable of causing equipment operation and erroneous status information. In addition, the testable l

SSLM's would have no previous operating history to

! justify it as an improvement to system reliability.

Based up;n the above discussions, PSE&G does not consider in-situ testing as a viable or cost effective l method of improving plant reliability at this time.

3.6 AUTOKATED 862 SSLM BENCH TESTER PSE&G has obtained an "Automated" 862 SSLM bench tester capable of testing all possible logic combinations with the module in its field configured state, i.e. with the l

FPLA on board and the staple jumpers in their field positions. The Automated tester is designed to perform l the following tests:

l Page 11 l

l J

- Input Buffer Range Test (staple jumper position determination) '

- Memory Functional Test

- Logic Verification Test

- LED Operational Test

! - Buffer Input Pickup and Dropout Test

- Time Response Test

- Buffered Output Leakage Test

- Fixture Operational Verification (self test)

The Automated tester data base verification has been completed, and technician training in the use of the tester is presently scheduled to begin during May 1988, l

4.0

SUMMARY

/ CONCLUSIONS:

Based upon input from several of the programs discussed above, PSE&G has come to the conclusion that the potential exists to increase the reliability of the 862 SSLS by reducing the Relative Humidity of the ,

[

environment in which the system operates. While the '

present failure rate is not considered to be excessive, PSE&G has begun designing modifications which will

maintain the Lower Equipment Control Room Relative l Humidity between 20% and 60%. It is anticipated that these modifications will reduce the average SSLM failure rate to approximately 1.1 failures per million hours of service. This estimate assumes that the failure rates experienced in the "low" humidity months can be maintained throughout the year. Refer to Attachment il for "in-house" SSLM failures.

This report is provided as part of a commitment which was required to demonstrate that 862 SSLM failures did not exceed 5% per year of the total population (2258 modules). Attachment #1 clearly indicates that the percent of SSLM failures over the last two year period was well below the 5% value.

\

Page 12 i

J

As a result of the effort discussed above, we have become aware of failure rate information which we feel permits a better understanding of normal failure rates for this type of device. IEEE Standard 500-1984, "IEEE Guide to the collection and Presentation of Electrical, Electronic, Sensing Component and Mechanical Equipment Reliability Data for Nuclear-Power Generating Stations" provides a recommended failure rate of 1.19 failures per million hours of service for "Solid State Computation Modules" (page 721). While the Hope Creek SSLM failure rate is presently above the recommended value, the values do not differ significantly, indicating that the reliability of the 862 system is consistent with comparable solid state Nuclear equipment.

5.0 SIGNATURES Prepared By : NbM/V CognigantEngineer Date : 3-El-88 Reviewed By : >

Hope Creek /IEC Group Supervisor Date : bM'N Approved By :

Nuclear Electrical Engirfeerisg Manager Date : 3'N'N l

Page 13 l

l

_ - .- - - --- , - . - - - - - - ~

BAILEY LOGIC MODULE FAILURES 10 For the period - March 86 thru Feb 88 10 '

l I

9 - .

\

8 -

l\

7 7 7 J n' 6 4 3 6 - d a >

l

' S 5 5 O k Z 6

5 -\ 4 \

1 4 k E

m m 4 -

l 1 \  ! Z 2 \ H z ,3 3 4 k  !

3 - t

_a 2 - \ 2 1

0 - . i i i i i i i i i iiiiiiiiiii MARAPR MAYJUN JLY AUGSEP OCTNOVDEC JAN FEB MARAPR MAYJUN JLY AUGSEP OCTNOVDEC JAN FEB

+ INSERVICE FAILURES i

L . _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ ___

, ,A9EACHMENT 2 COMPONENT FAILURE ANALYSIS Attached are reports OR-5106-E93-75, "Functional Verification 7 Report 862 Logic Module" and OR-5106-E93-75. Add-1, "Component Failure Analysis Report - 862 Logic Module" prepared by Bailey Controls Company.

Report OR-5106-E93-7$ documents the results of tests performed on 34 logic-modules to identify the failure mechanism of those

) modules. Report OR-5106-E93-75-ADDI documents results of tests performed to identify the failure mechanisms of solid state components associated with the non-conforming modules discussed previously.

These reports satisfy PSE&G's commitment to have Bailey perform a failure analysis on failed SSLMs as documented in PSE&G 1etter

> NLR-N86142 dated October 3, 1986.

1

)

\

b l

3

)