View source for B13235, Forwards Addl Info to Support Completion of NRC Review of Reactor Protection Sys Upgrade Project.Rev 0 to IC-CALC-88-001, Connecticut Yankee Instrument & Controls Calculations:Dynamic Response of Low Pressurizer... Encl

{{Adams
| number = ML20246M621
| issue date = 05/05/1989
| title = Forwards Addl Info to Support Completion of NRC Review of Reactor Protection Sys Upgrade Project.Rev 0 to IC-CALC-88-001, Connecticut Yankee Instrument & Controls Calculations:Dynamic Response of Low Pressurizer... Encl
| author name = Mroczka E, Sears C
| author affiliation = CONNECTICUT YANKEE ATOMIC POWER CO., NORTHEAST UTILITIES
| addressee name = 
| addressee affiliation = NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM)
| docket = 05000213
| license number = 
| contact person = 
| document report number = B13235, NUDOCS 8905190201
| package number = ML20246M626
| document type = CORRESPONDENCE-LETTERS, INCOMING CORRESPONDENCE, UTILITY TO NRC
| page count = 18
}}

=Text=
{{#Wiki_filter:..
1        .
           .C.
* NORTHEAST UTIRJTIES                                                o.nor.i Omc.. . s.io.n si,..t. so,iin. Conn.ciicui
                             '   '*
* P.O. BOX 270 k     '   J  .
                                        '"[*"
[*omw,u. . "u,(c,a,7,,".,4ao.co,,o'''          _**            HARTFORD, CONNECTICUT 06141-0270 (203) 665-5000 May 5, 1989 Docket No. 50-213 B13235 Re:    ISAP Topic No. 2.04 U.S. Nuclear Regulatory Commission                                                                                                     I Attention: Document Control Desk Washington, DC 20555                                                                                                                   )
i Gentlemen:
Haddam Neck Plant Additional Information Reactor Protection System Voorade On February 14 through 17, 1989, the NRC Staff met with the Connecticut Yankee Atomic Power Company (CYAPCO) and the Foxboro Company to review the upgrade of the reactor protection system (RPS) at the Haddam Neck Plant performed during the 1987 refueling outage. At the conclusion of the audit, the NRC Staff had identified a number of areas for which they desired additional information.
Accordingly, CYAPC0 is hereby providing the attached additional information to support completion of the NRC Staff review of the RPS upgrade project.
During the conduct of the audit, the NRC Staff stated (l) their position that because, in the Staff's words, of a " dramatic change in technology," the RPS upgrade project should not have been implemented under 10CFR50.59. It remains the position of CYAPC0 that the RPS upgrade work was properly performed under 10CFR50.59. This regulation expressly states that nuclear plant modifications can be made without prior Commission approval if (1) there are no unreviewed safety questions associated with the change (as specifically defined in 10CFR50.59) or (2) there are no technical specification changes necessary to facilitate the changes.           CYAPC0 previously had reviewed the RPS upgrade                                                      >
project and concluded that, as detailed in 10CFR50.59, these changes did not constitute an unreviewed safety question. This issue of project complexity is not discussed in 10CFR50.59 and to our knowledge has never been interpreted as a factor of consideration in determining whether a change constitutes an                                                                ,
unreviewed safety question.         Although a portion of the RPS upgrade involved a                                                    l change to the technical specifications, this change was approved by the NRC                                                             )
I i                 (1)    A. B. Wang letter to Connecticut Yankee Atomic Power Company, " Summary of                                                      i i                        February 14-17, 1989 Audit of Reactor Protection System Upgrade 50.59 Review," dated March 8, 1989.                                                                                                   l l
l 8905190201 890505              '
l PDR       ADOCK 05000213                                                                                                             '
P                       PNV ~                                                                                           ,f

$   =
4 U.S. Nuclear Regulatory Commission B13235/Page 2                                                                                                                                                                    )
        'May 5, 1989                                                                                                                                                                      )
l Staff in a license amendment (2) prior to the work being accomplished. In addition, these technical specification changes were only necessary. to reflect the fact that the RPS upgrade was an improvement over the previous system.                                                                                          In fact, most of the upgraded RPS could have been placed in service without the                                                                                                    j changes to the technical specifications. As such, CYAPC0 maintains that both the " spirit" and " letter" of 10CFR50.59 were followed for the RPS upgrade                                                                                                      j project.
In addition, CYAPC0 has endeavored to provide the NRC Staff with information regarding the details of this project.                                       CYAPC0 has provided initial submit-tals, responded to requests for additional information and met with the Staff on previous cccasions to keep you apprised of this project.
If you have any additional questions, please contact us.
Very truly yours, CONNECTICUT YANKEE ATOMIC POWER COMPANY b.U.D                                           V E. J. Mroczka Senior Vice President By: C. F. Sears Vice President Attachment cc: W. T. Russell, Region I Administrator A. B. Wang, NRC Project Manager, Haddam Neck Plant J. T. Shedlosky, Senior Resident Inspector, Haddam Neck Plant (2)   F. M. Akstulewicz, Jr. letter to E. J. Mroczka, Technical Specifications for Cycle 15 Operation (TAC No. 65479), dated November 12, 1987.
_ __ _j

i      c.
0 Docket No. 50-211 813235 Attachment Haddam Neck Plant Additional Information RPS Upgrade Project l-l                                               May 1989 f

i     :.
6
     .,                                                 RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 IIRM 1 Several' errors were. identified by Foxboro in the software. There seems to be I no paper trail that concludes these items were corrected. The staff believes an independent review should be performed to verify that all identified software errors were corrected. The review should include at least the four-    3 modules being used by CYAPC0 but can include all modules produced by Foxboro. i

===Response===
An independent review, to verify that all software errors found and documented during control block code walkthroughs have been corrected, was completed March 24, 1989, for the following four block types used by CTAPCO: GATE, LLAG, ALRM, CALC. All software errors identified were found to be corrected.
The review was conducted by Hans Fritschi, Sr. Engineer, D345, Corporate Product Engineering, Foxboro Co., and Valter White, Software Quality Assurance Manager, C0A, Foxboro Co.
                                   . Verification has been documented on an item-by-item basis and filed in the SPEC 200 MICR0 Product file within Corporate Product Engineering.
All other control block code walkthrough issues found and documented during product development will be reviewed to insure each has been resolved.
Targeted completion is mid-year 1989.
I                                           4
                       ,,                                                                                 RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM 2 The staff needs CYAPC0 to document the software reliability program being used by Foxboro.

===Response===
l See Foxboro's Software Reliability Program (Attachment 1).
l
                                                                                                                                                            - _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - . - . - _ _ _ - . - - - - - - -               -  --                                                    - ------u

i                         .
A
                           ,                                    RESP 0FSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM 3 The staff needs CYAPC0 to state that the new system is functionally equivalent to the original system (comparison of digital curves versus analog curves).

===Response===
The new system is functionally equivalent to the original system. The dynamics     l associated with the Variable Low Pressure Scram (VLPS) calculator were the only safety system dynamics which were replaced with a digital equivalent. The          i dynamic response of the new digital system was demonstrated to be functionally equivalent to the original analog system by a verification of the digital system's response to a step input. Calculation IC-CALC-88-001 was performed to determine the expected step response of the analog system using the Laplace Transform method. A step input was then applied to the digital system to demonstrate equivalence between the actual and expected response. This testing is performed under approved station surveillance procedures which specify the test methodology and acceptance criteria. Calculation IC-CALC-88-001, along with sample expected and actual response,:urves, is attached (Attachment 2).
l l
1
                                                                                                                                 )
i i
I

s
      ,                                           RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM 4
                                  'CYAPC0 needs to. demonstrate that the EMI' testing by Foxboro bounds the EMI spectrum at the Haddam Neck site.

===Response===
No quantitive analysis or testing was performed to determine the Electromag-netic Interference (EMI) spectrum at the Haddam Neck site.- From a qualitative.
standpoint, however, the EMI spectrum was considered to be low since the equipment is installed in the Main Control Room away from high power equipment and cabling. In addition, administrative controls prohibit the entry of EMI
    ~
sources, such as portable handheld radios, into the Main Control Room. The EMI testing by Foxboro was assumed to bound the expected low EMI spectrum at the Haddam Neck site. Subsequent testing performed by the Foxboro Company, for Swedish State Power, has demonstrated even lover EMI susceptibility than initially documented (Reference Foxboro Test Report No. 88-1033a, dated November 30, 1988). This testing provides further assurance that the EMI spectrum at the Haddam Neck site is bounded, i
l f
1

t            c s
          .-;                               RESPONSES'TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM 5 CYAPC0 needs'to document'that the fault testing of L2CR (relay card) was by a transformer at 20 Amps or greater.

===Response===
Fault testing of the 2A0-L2C-R contact output isolator was performed by Foxboro.
Personnel at the facilities of-Acton Environmental Testing Corporation (AETC)
                            'in Acton, Massachusetts, and the test results were documented in the Foxboro                    i Company Corporate Quality Assurance Laboratory Test Report No. 00AAA20, Part-1.
However, the transformer used to supply the 600 VAC Test Voltage was not-documented. The un!t belongs to AETC and was borrowed from their equipment pool to perform the test. Later, a decision was made to not list it in the test equipment section of the test report, since the unit belongs to AETC, not Foxboro.
It is now obvious that this piece of equipment is important, and so the following pertinent information is submitted, which was obtained directly from the data label an the unit:
Manufacturer      Description         Model    Serial No. Output Rating                        l l
Strong       Portable AC        60000-2     43632      630 Amperes Power Supply                             Vhen Operating                        .
                                                                                                                           .1' On The 600 VAC Range Note: This unit is AETC Inventory No. PA-501-72.

1,                o
                                    ..                                                                                RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM 6 l
CYAPC0 needs to document that the seismic test performed for the L2CR envelopes the seismic spectrum at the Haddam Neck site.
                                                                                                                                                                                                                                                                                        ]

===Response===
4 A seismic qualification review and independent review was performed by                                                                                                                                                                        ;
Northeast Utilities Service Company's Generation Engineerir.g Mechanics group prior to project implementation which determined that the Foxboro Contact Output Isolator's (2AO-L2C-R) seismic qualification was acceptable. In response to your concern, an additional seismic qualification review and independent review was performed recently which also concluded that the seismic test performed envelopes the seismic spectrum at the Haddam Neck site.
Although the equipment did not meet the target Test Response Spectra (TRS)-
acceptance criteria at the Safe Shutdown Earthquake (SSE) level, the equipment is acceptable since the target TRS acceptance criteria at the Operating Basis Earthquake (OBE) level was met which envelopes the Required Response Spectra (RRS) for the SSE conditions at Haddam Neck (Attachment 3).
1 l

   ?   ..
     ,                                     RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 n                                                                                                                                             .;
ITEM 7                                                                                                                 i When the watchdog timer tries to reset-following a time-out does the trip contact also reset?

===Response===
The trip contact vill not reset following a watchdog timeout. Following a vatchdog timeout, the CCC Card vill go immediately to the FAIL mode. The card outputs will go to their jumper configured flunk state.                                       If the watchdog timer does reset, the card vill go to the the ERROR STANDBY. mode and the card outputs vill remain in their jumper configured flunk state. Recovery of the CCC Card failure from the ERROR STANDBY or FAIL mode can be attempted only via the configurator or FOXNET. (Reference HI 280-300 page 10.).
A watchdog timer is provided on the CCC Card in the event the CPU malfunctions.
If the watchdog timer can not be reset by the CPU within 250ns, the CCC Card will be driven to the FAIL mode, the card outputs will go to their jumper con-figured flunk state and the CPU vill attempt a vectored restart procedure. If the CPU can not successfully complete the initialization procedure, the card vill stay in the FAIL mode. If the CPU completes the initialization procedure but hangs up again, the watchdog timer vill timeout again, and another initialization vill be attempted. This sequence vill continue indefinitely.
                       -If the error that caused the watchdog timeout was transient, the controller should restart successfully and vill be driven to the ERROR STANDBY mode. The card outputs will remain in their flunk state while the card is in the FAIL or ERROR STANDBY mode.
1

; .3           .                  ,
J r
               .,                                                                            u ,,
     .                                        RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989                                                             )
                                                                                                                                                              ,i J1-                ITEM 8                                                                                                                                   )
CYAPC0 should justify the use of any-nest that uses more cards than'the Foxboro                                                     '
recommended three cards per nest.

===Response===
No safety-related application utilizes a nest that uses more than the Foxboro recommended threeLSPEC 200. Micro cards per nest. A single non-safety related, application was found to utilize a nest that uses four SPEC 200 Micro cards.
Through discussions with Foxboro, this recommendation was found to be based on conservative temperature calculations performed to qualify the equipment over its entire range of ambient temperatures.                                         Since this application is'non-safety-related, and since the equipment is installed in the temperature
                    . controlled (50-110'F, normally 75"F) Main Control Room, no challenge to safety isLidentified to necessitate immediate action.
I
                                                                                                                                                                                                                                 - _ _ _ - _ _ _ _--___--       A

c
    ,                                     RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 ITEM-9 Vas a failure modes and effects analysis performed or updated for the new RPS?

===Response===
In fulf311 ment of.the recommendations of Regulatory Guide 1.70, Standard Format and Content of Safety Analysis Reports for Nuclear Power Plants, Section 7.2.2, an analysis of the original Reactor Protection System (RPS) was performed as documented in the Connecticut Yankee Final Safety Analysis Report, Section
                        -7.2.2. This analysis was performed on'the system level and was found to bound the modernization effort since the new RPS is functionally equivalent to the original-system. A detailed Failure Mode and Effects Analysis (FMEA), as described in'IEEE 352, Guide for General Principals of Reliability Analysis of Nuclear Power Generating Station Protection Systems, however, was not performed on either the original or the replacement system. Instead, independent design reviews, preoperational testing, and safety evaluations document the failure modes of the replacement RPS and.the acceptability of their consequences to ensure that safety is not compromised. In addition, a comparison of new equipment failure probabilities to those of old equipment was performed in accordance with IEEE 577, Standard Requirements for Reliability Analysis in the Design and Operation of Safety Systems for Nuclear Power Generating Stations, which concluded that the modernization would likely enhance the reliability of the system and thus have a potential positive effect on plant availability.
: t. ; ., . ..
     .c                                    RESPONSES TO NRC AUDIT ITEMS OF FEBRUARY 14-17, 1989 t
ITEM 10 Provide the root cause analysis of the failed card at the Haddam Neck site.

===Response===
A root cause analysis of the failed SPEC 200 Micro card at the Haddam Neck site is not available since'the card was not: returned for repair as originally
                       -assumed. The failed card was apparently lost during its preparation for shipment to..Foxboro and efforts to locate it have failed. The work order which replaced the failed card identified its failure as a Vatchdog Timer Timeout.
                   .However, no root cause was identified.
1

                 . .                                                                      RESPONSES TO letC AUDIT ITEMS'0F FEBRUARY 14-17, 1989 s
ATTACHMENT 1

t FOXBOR0'S SOFTWARE RELIABILITY PROGRAM i
Producing reliable software is of paramount importance in the process control arena.
errors.
Yet virtually all new software contains The large industrial process control systems currently being designed and built have more than one million lines of source code. If, as experts estimate, there are initially between 0.5 and ten defects per KSLOC (thousand source lines of I           code), this is a potentially serious situation.         A quantifiable measure of goodness is needed to demonstrate that the final code is reliable, the defects having been successfully removed.
Dozens of probabilistic reliability models have been suggested and described over the past fifteen years or so. They have their place, but most of them are unrealistic in their assumptions, and thus work only part of the time.         This adds to the " spooky" aura surrounding the entire software reliability field.
The Foxboro Company has been using a simple reliability growth mode.1 for five years.
This model is called the Duane Growth Model after it's originator, J. T. Duane. This is a very simple mathematical improvement effortmodel,   and is applicable as long as reliability continues. One may graphically obtain the answers to key questions, such as:
* Where are we?
* Where are we going?
* How fast are we getting there?
Duane's model is mathematically expressed as:
E            -x
                    ---   =  KT T
Where:    E = Total defects discovered T = Tote 1 testing time (CPU time)
K = an equation constant x = Growth rate The mathematical details and application criteria may be found in the reference papers listed at the end.
For practicality, two separate databases are set up. The first, referred to as the "buglog", contains records of defects discovered, the date of discovery, system symptoms, the offending software module, and so on. Of course, ALL defects discovered must be included in this buglog.         The second contains records of the number of CPU test hours expended on a oaily basis over all active independent copies of the software under test.
In practice, one takes the available defect and test hours data and plots the information in the form of a log-log plot of the Cumulative Failure Rate (E/T) for each defect versus the accumulated test time (T) at the time that defect was discovered.
The method is simple and easily adapted to comput_er automated plotting, and the results are accurate.          A typical plot is shown in the following figure. Once the plot is        constructed, and reliability

            +    ,
i growth is verified (cos tha rafaranca articles), ona may then           !
calculate the MTTF (Mean Time To Failure, or the time to the next defect discovery) from the partial derivative value at the last point {
on the graph. In addition, simple algebra will allow projection     i into the future, to predict how many defects will be discovered in, say, the next 2000 hours of unit testing. This is useful in determining how much additional testing is required to verify the system software reliability before a release is contemplated.
Currently, Foxboro strives for a minimum predicted MTTF of 2000 hours to the next critical defect prior to release.
Two other tools are used at Foxboro:      The Discovery Plot and the Pareto Profile. Both are discussed at length in the first of the reference papers listed (Noon - 1986].
A final point is the relationship of test hours to time in the real world. When Foxboro personnel do system software testing, either by specific test cases or by random testing, the expended hours are much more intense than normal operating hours for a system would be. The tests are designed to traverse every possible path through the software, and thus stress the design of the system in unusual ways.       This testing affords the maximum opportunity for uncovering defects that may be present in the code. In normal industrial operation, the situation is different.
Much of the time that a system is running, the operating system is waiting in an ' Idle loop', looking for a command. This time is essentially of zero value as exercise time, because the same path is being repetitively transverses over and over. Similarly, much of the normal operational processing covers only a small number of paths. Usually the same menu structure is traversed, the same control blocks remain configured, and the process data limits are roughly the same. Thus, it is unusual when a different path through the software is travelled.
This phenomenon has been considered by software development engineers and quality assurance personnel in the recent past.
The general agreement was that the combination of factors resulted in a net expansion of the MTTF value derived from testing of system software as described above by about a factor of 50. These are empirical figures, based on personal judgements of field experience as we know it.
The net result of this is that a testing MTTF of 2000 unit hours translates into on estimated operating MTTF of about 100,000 unit hours, or about 11.5 years. The individual unit failure probability is then about ten to the minus fifth.
REFERENCES Noon, David W.,     " Practical Software Reliability", Proceedings of IFAC Workshop on Instrumentation Systems for Safeguarding and Control, May 12,1986.
O'Connor, Patrick, PRACTICAL RELIABILITY ENGINEERING, New York, John Wiley & Sons, 1983, pp. 203-209.
Codier, E. O.,    " Reliability Growth in Real Life", Proceedings of the IEEE Annual Symposium on Reliability, 1968, pp. 458-469.
i__-._________

    .p(                          .
7 g
0
             -              3                                                                                                             0 e                                                                                                             0 0
T         9                          .                                        .  .
                                                                                                                   ..                     0 O             .
                                                                                                                             .       . 1 L                                              .
y P
s H            r G
T            o W
                    .N.                        . . . .                                                                                        )
r            .
O  A                                                                                                                       0 r                       .
                                                                                        =.
0 sr R
G c         . 0
                                                                                                                                     .0(
C       .              .
1 e
E  _-        e            .                                                                     -
g.
l m
 ...-          N            a               . .                                         -
  .. . g       A  0         y i
T U         -
t D
s
                                                                                        = . .
e T
L  A                                                                                                   .
0l A  R                .
                                                                                                                                     . 0 o
                                                                                                                                     .      0l C                         . . .
                                                                                                          .           ..             . 1 o
I  R.                     . .
T O
P                         . . .                               .                  .
i                            . . ..               .                                     - .
Y R
T  R                            ..
A
                                            . .                                  1I                                   ..
                                                                                                . d. .             .  .
0 0
1
                                                    .                                   -                                         1
                                       -                                              1 1                                                                                            0 0                                           0 0                                                0                                           0 e        _-
                                                                ,V      .t ng-F   o.o NuLouLg
                                                                                    -   t a._o.
                                                                                                                         .F g
r

RESPONSES TO NRC AUDIT ITEMS f F FEBRUARY 14-17, 1989 ATT ACHMENT 2 1
l L-.                                                                   -- - - ._-____---__-_________2}}