ML20012E502
| ML20012E502 | |
| Person / Time | |
|---|---|
| Site: | Haddam Neck File:Connecticut Yankee Atomic Power Co icon.png |
| Issue date: | 03/21/1990 |
| From: | Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML20012E500 | List: |
| References | |
| NUDOCS 9004050276 | |
| Download: ML20012E502 (31) | |
Text
1 s
. '.* m.,9 e
[( ' e umTap sTAtas l
NUCLEAR REGULATORY COMMitSION n
/)!
i*
5a
=Asmweim, s. c. mm s%
l j
ENCLOSURE :.
)
SAFt1T tyA:.uAul0N BY THE OFFICE Or KL' CLEAT REACTOR REGULATION kEACTOR FFDT[CTION 5Y5 TEM UFERAD: IFMA5EUNL)
C0liNECTICUT YAKKEE ATOMIC POK:R CmrANY HADDAM NECK PLANT DOCKET ND. 50.Z13 i
TABLE OF CONTENTS l
1.0 1HTRODUCTION l.
2.0 OBJECT!YES j
t j
3.0 TECHNICAL APPROACH TO THE EVALUATION I
3.1 GENERAL 3.? APPROACH TO HARDilARE AttD SYSTEtts ASSESSMENT 3.3 ArrROACH TO SOFTWARE ASSESSMENT f
4.0 DESCRIFTIO!: CT SYSTEM UNDER REY!EW A.1 DES!C!: BAS!$ DESCRIPTION i
4.2 FUNCTIONAL /SYSTEll DESCRIPTION 4.2 I;ARDWARE DESCRIPTION 4.A SOFTWARE DESCRIFTION 5.0 TEthillCAL EVALUATION 5.1 SYSTEMS AND HARDWARE ASSESSMENT 5.1.1 Design Basis
.1 Criteria Confonnance
.2 Functional Requirements l
,3 Adequacy of Licensee's Safety Analysis 5.1.2 Failure Modes and Effects
.1 Detectability of Failures
.2 Special Failure Modes t
.3 Acceptance Tcstir>g l
$$f0$$bbk bO ' f I
i P
J
l 5.1.3 Effects of Retrofit of New Equipment to Existing Systems
.1 Design Basil Surge levels on Power Connections and I
Electromagnetic Interference (EMI) j
.2 Suitability of Existing Cable for New Instrumentation 1
5.2 SOFTWARE ASSESSMENT 5.2.1 Criteria
.1 Verification
.2 Independence
{
.3 Testing I
4 Valication
.5 Discrepancy
.6 Configuration Management 5.2.2 VI.Y Plan 5.2.3 Design Process 5.2.4 Implementation Process 5.2.5 Testing
.1 Verification Testing
.2 Validation Plan 5.2.6 Configuration Management I
5.2.7 Conclusions
6.0 CONCLUSION
S C.1 SYSTEMS AND HARDWARE ASSESSMENT CONCLUSIONS 6.1.1 General 6.1.2 Unresolved items 6.2 SOFTWAPE ASSESSMENT CONCLUSIONS 6.3 10 CFR 50.59 Conclusion I
f
j 1.0 It;TF0DtICT!hN Connecticut Yankee Atomic Power Co. (CY) has upgraded portions of the reactor l
protection system (RPS) and the NS$$ control system of the Haddam Neck Plant
- HtiP). CY has implemented Phase I of the up'rede on the basis of a 10CFR50.59 review, ire the entire retetor protection and control system *; this additional and the system is now operational. Ibase !! of the HNP upgrade will t
" modern effort is planned for a future outage and is not in the scope of this evaluation.
CY concluded in their 10CFR50.59 review that the upgrade was essentially a replacement in. functional kind of obsolete RPS and NS$$ control system analog i
modules with modern microprocessor based modules. The modification includes several parameters of the RPS and involves interfaces with major control j
systers. The 10CFR50.59 review concluded that there'are ne significant changes in RPS or control system functional requirements / configurations, and no degradation with respect to the original and current plant design basis (for i
example, no different failure modes and effects have been identified by CY) CY j
concluded there are no unreviewed safety questions resulting from the change.
The unique feature of this upgrade is the use of licensee.configurable. fic l
microprocessor based, protection and control system modules.
The speci guipment and configuration 9f this upgrade have not previously been reviewed by the staff and therefore ho direct corparison can be made to a previously i
approved desigr.
l 2.0 OyECTIVES The object ne of this SER is to determine the acceptability of the Phase One l
PPS upgrade. This SER will also evaluate the Foxboro equipment and design
(
methodologies to reduce the need for additional audits for Phase Two of the RPS upgrade. This SER will evaluate the applicability of using 10CFR50.59 to make this nodification without prior NRC approval.
Mtendant ley issues addressed in the SER are the specific design bases for the rew systen end the original system; potential hardware vulnerabilities and susceptibilitiestsoftwaredevelopment,verificationandvalidation(Y&Y),
i configurttien managerentt potential control / protection systems interactionst systtm failure modes and effects; reliability and the developmental and operational history of this equipment.
l i
I r
4 4
--,n-
-.---,w
.t-2.0 TECPHICAL ArrROACH TO THE EVALUATION 3.1 GENERAL The licensee was requested to demonstrate compliance to the appropriate criteria and demonstrate that the use of microprocessor based equipment will not degrade the reliability of the RPS relative to the plant design basis.
l Pertinent criteria used (in addition to the Standard Review Plan (SRP)) as a basis for the audit was Regulatory Guide (RG) 1.152 ' Criteria for Programable l
Digital Computer System Sof tware in Safety Related dystems of Nuclear Power Plants" which endorse ANSI /IEEE.ANS 7 4.3.21982 ' Application Criteria for Programmable Digital Computer Systems in Safety Systems of Nuclear Power i
Generating Station'. The audit reviewed the following major topics which are important to safety and required licensee / vendor support in providin appropriate documentation and personal interface wita the reviewers:g j
t 1.
Assessment of functional equivalence or improvement of the upgrade relative to the original design basis.
l 4
P.
Assessmentofthevendor'sdesign(FunctionalRequirements&
Specifications)andVerificationandValidationprocesses.
3.
Assessrent of licensee and vender procedures and methodologies for the transformation of functional requirements to the software configuration that implements these requirements.
4 Assessment of CY's development of the design modification with respect to the design basis.
l S.
Assest.nent of CY's configuration manegerent process for the new
[
design.
r These majcr topics are addressed from hardware, sof tware, and systems l
trgineering perspectives.
In executing the audit, a " thread concept" evaluation trproach was used whereby a single functional (sensor thru ectu6 tor)requirementwasfoIlowedfromdesignthroughimplementation,through i
verification and validation testing, and through any retesting required when a i
failure is encountered. Where necessary, additional signals were examined to I
sufficiently cover the hardware, software, and systems evaluation criteria.
l 3.2 AFPROACH TO HARDWARE AND SYSTEMS ASSESSMENT The licensee was asled to demonstrate that all hardware / system vulnerabilities and susceptibilities that have potentially greater significance in a digital i
system than in an analog system have been addressed in the design.
Particular attention was requested for interfaces of the new digital equipment with old
[
equipment or circuits, including power sources.
It is important that failure t
modes and effects that may be significently different for digital hardware and systemt be adequately addressed by the licensee. As part of the review the licensee and vendor also had the opportunity to provide evidence of the enhancements provided by the new equipment such as higher reliability and more accurate control such as a reduction in instrunent setpoint drift.
i i
l g
3 Areas of potential vulnerability /$usceptibility addressed by the licensee / vendor and assessed in the audit include the following:
1.
Temperature environment (particularly within cabinets) and humidity.
2.
Electrical, voltage, frequency, surge Withstand Capability ($WCl credible faults; system tolerance to power surges and the use of existing inverters to supply power to modern logic elements.
3.
Electromagneticinterference(EMI)includingRadioFrequency Interference (RFI) and noise propagation via power lines to the new equipment.
Susceptibility of the new equipment to radiated EMI and RFl.
4 Failure Modes and Effects Detectability of failures and recovery, 4.
b.
Special failure modes (for example, system stall, timing errors / instability) and the effects on frequency and severity of transients.
5.
Technology Upgrade comparisons Comparison of reliability / availability of new a.
equipment to those of the original equipment.
b.
Identification of failure mode differences between new and old equipment; graceful degradation features.
6.
Independence i
a.
Physical and electrical separation and barriers.
b.
Isolation devices and their application, c.
Independence of various software functions.
7.
Environmentalqualification(10CFR50.49) i 8.
Seismic qualification (Regulatory Guide 1.100)
To support the audit, the licensee was requested to provide a clear and sufficient definition of the design change to enable a comparison of the original reactor trip system (before the upgrade) and the current design (after the upgrade). This was to include the following documentation:
1.
A functional block diagram of the reactor protection system (before and af ter the upgrade).
2.
A hardware block diagram of the reactor protection system including interfaces (before and af ter the upgrade).
l
[
l i-('
1 4
I s
3.
Clear definition, on the above drawings, of protection / control system boundaries; protection division (channel and train) boundaries power sources / supplies: and isolation devices or other methods of assuring independence that are used at these interfaces.
4 The modification package, supporting design documentation, and safety evaluations.
5.
Identification of the standards, criteria, specifications, calculations analyses and tests that address the hardware attributesofinterest. The criteria assumptions and methodologyusedtoestablishthedesIgnbasisfor,the electrica environment and EMI environment, and demonstration l
of compliance to the design basis were of special interest. Also, the s
licensee was asked to provide or have available for audit the following:
1 1.
Factory acceptance test reports i
2.
Site acceptance test reports 3.
Malutenance logs for the system l
4 Operating and maintenance procedures i
i 5.
Test procedures / reports for periodic testing of the system 6.
Data on self-diagnostic failures 3.3 APPROACH TO SOFTWARE A$$ES$ MENT The inclusion of microprocessors and their concomitant software in the RPS
{
marks a significant departure from the original analog electronic design.
While the transition to digital systems may provide significant performance and 56fety advantages, it may also introduce issues and concerns that have not been j
enccuntered previously anc have not undergone a thorough safety review.
For the software assessment the licensee / vendor must demonstrate that the software is in accordance with the functional design, and that the t
impleinentation process will result in reliable sof tware. The licensee must also cemonstrate that an adequate program is in place to assure that the reliability of the sof tware is maintained for the entire life cycle. The licensee must consider revisions to the software required to correct errors or provide i
adcitional capabilities. The licensee must also provide adequate controls on the user configurable software.
To assess these points of concern, the software audit focused on three main i
areas:
1)
The design, development and operational history of the vendor's software.
2)
The procedures and methodologies in use by the licensee / vendor to ensure that the vendor supplied hardware and software control blocks are functionally equivalent to the design basis for the RPS.
t
3)
The procedures and V6V processes used ty the Itcenste/ vendor to control the user configurable software elements.
To support the sof tware assessment, the licensee / vendor was asked to make available the following information:
1.
Description of the design oevelopwent and operational history of the vencor's sof tware components.
Description of the design and software development activities, 4.
b.
The scope and results of static and dynamic tests for the software.
A description of the acceptance testing performed and the c.
results of the testing, d.
Documentation of all modifications perfonned since release.
A description of the FROM
- burn in" process and what procedures e.
are in place to ensure that the finished PROM is correct.
2.
A description of the procedures and methodology used by the licensee to ensure that the functier.a1 design ba61s is implerented, The functional specification for the system and the derivation s.
of the control block configuration, b.
The procedures and nethods used to ensure that the control block configuration will implement the control logic as specified.
A description of the licensae's user configurable software c.
manegement program, The results of testing the completed system, the list of errors, c.
how the errors were detected and corrected.
3.
Description of the procedures and V4V processes used to control the configuration of software for the control blocks.
A description of the vendor's verification activities for the a.
control block sof tware and the formal Y&Y plan, b.
The results of verificaticn activities during the develo >nents characttrite the errors found by type and percentage of tie total errors, The results of validation activities during testingt c.
characterire the errors found by type and percentage of total errors.
-6 d.
Description of the steps taken to ensure the incependence of the verifier and his recourse in the event that discrepancies or errors are found, j
4.0 DESCRIPTION
OF $YSTEM UNDER REVl[W 4.1 DE$lh BA$l$ DESCRIPTION The original functional requirements for the RPS are contained in Westinghouse DocumentCYW300(1965). The licensee issued specification $P.EE.!!6 in 1986
)
for procurement of the upgrade modules, and Plant Desi
$(1wasissutoin1986toimplement.thepodification.gnChangeRecord(PDCR)
Toxboro issued four documents in 1988 that described conformance to Ittt.603 and ![t[/ANS 7 4.3.2.
j i
i The staff revined the standards and criteria listed by the licensee and/or vender in the various design documents. Design attributes of special interest i
i to computer based systems that wer6 not addressed in depth in the above documents include EMI and $WC.
i 4.2 PUNCT10NAL/$YSTEM DESCRIPTION t
l The changes are intended to mocernire the Reactor Control and Protection System i
instrumentation by re Pressurizer Pressure. placing a portion of the original equipment supplied for Pressuriter Level, RCS Delta Temperature, RCS Average
[
Temperature Flow Control.RCS Wide Range Temperature. High Pressure Steam Dump and Charging The instrumentation affected by the change are i
a)
Replacement of the transmitters and Main Control Board mountedequipmentforPressurizerPressure(LoopsP4011, P4012andP4C1-3),
i b)
Replacement of the transmitters and Main Control Board mountedequipmentforthePressurizerLevel(LoopsL401-1, j
L4012and401-3).
l c)
Replacement of the transmitter and Main Control Board mounted equipment which makes up the First $tage Turbine Pressure (LoopP1204). This loop is upgraded to Category 1 Class 1[.
i i
d)
Replacement of the transmitter and Main Control Board mounted equipment which consists of RCS Delta Temperature, i
RCS Average Temperature, and RCS Wide Range Temperature.
i e)
The addition of a fourth channel of Pressurizer Pressure (Loop P4014). The sedition of circuitry to provide a i
fourth channel of Yariable Low Pressure Reactor Trip which l
changes the reactor trip logic from 2 out of 3 to 2 out of 4.
i i
l f)
Replacement of the transmitter and Main Control Board mounted equipment which makes up the main steam header pressure loop (P1203) and the High Pressure Steam Dump valve control system.
9)
Replacement of the transmitter and Main Control Board mounted equipment which ewkes up the Charging Flow Loop (F110andF110A).TheaeditionofaFlowcontroller (FIC 110A) to provide redundant charping flow control in additiontotheexistingflowControler(FIC-110).
h)
Replacement of the circuitry and Main Control Board mounted touipment to proviet Rod Direction and Rod Speed Control inputs.
1)
Replacement of the circuitry to calculate the Pressurizer Level Prograrsner remote setpoint.
This includes replacing the Taylor Reccrder/ controllers and the age degraded wiring.
i j)
Replacement of the aged wiring and hardware that provides Low Temperature Overpressurization interlocks.
6,."
,,n
.N k)
Relocation of the RCS Wide Range Pressure Recorders to the....
reer of the Main Control Board, j;
1)
The sedition of Narrow Range Cold Leg Temperatsre indication (TI 411B, 4216, and 4418) to the Main Control Board, m)
Replacement of the Main Control Board mounted RW5T Level indicators.
These changes will affect the operation of the following systems:
Rod Control
-pressurizer pressure and level control
-RCS charging flow control
-Steam bypass control
-Post trip feedwater control The functions of the RCS operation affected are Yariablelowpressure(VLP)
-High pressurizer level High pressurizer pressure To implement these functions the licensee has procured a series of microprocessorbasedcontrol, modules (FoxboroSpec200 Micro)thathavebeen installed in existing Foxboro Spec 200 racks at HNP. These modules are intended to functionally replace the ori that were installed in the control room.ginal equipment (Hagan) analog modules The replacements were made on a loop for loop basis, and were not installed as a distributed cigital network.
l 5
3
)
l l
An example of a distributed network is provided in NUREG 04g3. *A Defense in Depth and Diversity Assessment of the RESAR 414 Integrated Protection System" (March 1979).
In en integrated protection system the staff had concern that certain classes of failures which may i
designs may have a potential for serious consequences. be tolerable in other i
The failure of most concern is a consnon-mode failure of redundant elements.
A highly intelrated system with one microprocessor performing several functions may have tbe ability to propagate failures beyond the system where the failure originally While a distributed network has certain advantages and can be i
occurred.
designed so that it is acceptable to the staff, potential consnon-mode failures l
must be addressed in detail l
~
In a stand alone system such as used at Haddam Neck the input from the sensor i
is processed by a deeicated microprocessor with output to the process i
The level of functional diversity in the original design is trip /actuater.
maintained.
The stand alone approach provides a somewhat simpler method of t modernizing a system without as many cianges and dependencies as the i
distributec system.
1 4.3 HARDWARE DESCRIPTION l
The hardware is based upon the application of the Foxboro Spec 200 Micro control System. The system consists of control cards mounted within the i
existing cabinets and display stations located at the various control panels within the control room. The control card is a rack mounted microprocessor based unit which perfoms signal conditioning, regulatory control and logic l
control functions. The control functions 6re provided by the user configuration of up to 6 control blocks from a menu of 21 different types. Most i
block types have multi-functions available. The display stations are available in two types. The Continuous Display Station provides user interface to the ccnfigurable control blocks. The Discontinuous Display Station provides user interface to logic control blocks. These displays provide sequential paging to display more than one control blockh two nine digit alphanumeric readouts i
provide configurable loop tag ident'fication, digital readout of the block parameters in percent or engineering units and the ability to modify control block tuning parameters.
The system can be built as a stand alone or distributed system. The licensee's t
system is configured as stand-alone. Also the systems are constructed with the new hardware items applied in conjunction with the existing SPEC 200 products.
t The hardware items were tested to the requirements of IEEE 344 1975 and portions of IEEE 323 1974 Environmental specifications to provide compatibility with the control room environment were identi'ted for the I
r equipment. The units are puwered from 117 Vac vital bus power.
l 4.4 SOFTWARE DESCRIPTION The Spec 200 Hiero is a sof tware based control system that uses a digital card t
to implement a wide range of control structures and logic. The Spec 200 Micro
.g.
l softwarewasbasedongeneralfunctionalrecuirementsderivedfromthe cc11ective functions o<
approximately 100 existing Foxboro Spec 200 analog The sof tware is presented as 21 modules called Control Blocks control cards.
i that can be configured for specific applications by a menu driven set up process. The responses selected from the menu by the licensee or vendor fills l
out internal tables which then controls the operation of the specific control functtons and logic.
l On a single digital control card, an executable module for each of the 21 Processor Control Blocks are contained in the programable read only memory i
(PROM) and cannot be changed by the licensee, Up to 6 of these modules can be configured by the licensee at the site on any one control card to implement the i
required control logic.
The system is configured by using a personal computer.
i The configuration sof tware provides menu driven " fill in the form" displays.
These displays allow the control scheme to be configured or to modify the i
control blocks contained within the control card. Configuration is the process of selecting control block types and interconnecting these blocks to produce the desired control scheme.
Displays allow the monitoring of the system parameters to verify selected control schemes.
The Licensee's Phase 1 configuration uses only three of the available 21 Control Blocks for safety systems. The three blocks used are CALC, LLAG and ALRM.
In addition Phase 2 will use GATE. Additional blocks are used for non. safety applications.
i i
CALC provides facilities for executing simple arithmetic equations. Fourteen functions are available and can be combined with data inputs and intermediary results to form the equation.
I The format is similar to that used on programable
(
pocket calculators.
ALEM provides the facilities for setting alarm points for data inputs from the It includes absolute alarms, deviation alarms, rate alarns, output sensors.
alarms end output limiting.
I LLAG provides the facilities for setting lead lag compensation parameters. The Tee 8 is defined by the configurator as an amplification factor of the lag term, anc the lag is defined as a time constant.
GATE provides eight two. input lo Edivan algebra sand, or, vtc.) gic gates which allows configuration of various functions.
i 5.0 TECHillCAL EVAlt!ATION i
5.1 SYSTEMS AND HARDWARE ASSESSMENT For the systems and hardware assessment the staff reviewed the design basis, the f ailure modes and effects of retrofit of new equipment to existing systems.
i l
l
- L
I I i
5.1.1 Design Basis The cesign basis review included conformance to criteria, a review of the functional requirements and an evaluation of the adequacy of the licensee's safety analysis.
5.1.1.1 Criteria Conformance The principal criteria of interest in this evaluation were AH51/lEEE Ah5-7 4.3.2 198P, IEEE 279 1971 IEEE 603 1980 IEEE 384 1981, IEEE. 323 1974 and IEEE 344 1975. TheapplicabIlityofthesec,riteriatothe modification of interest were. collectively established by the UF$AR, the licensee's system specification $P EE-226 the system vendor's requirements conformance cocuments and the licensee's Plant Design Change Record (PDCR) 861.
Except for the original specification provided by the NS$$ vendor the highest level design document is the licenste's system specification.
ThIsdocument should stipulate all of the principal standards and criteria to which the system must be designed, implemented and tested; these requirements should be verifiable during the design, impleme,ntation, and testing process for the system. This document also should have captured all relevant requirements of the licenste's UFSAR and licensing commitments. The staff noted weaknesses in the licensee's system specification in that the standards referenced, although appropriate, were not sufficiently comprehensive in specifying requirements for electromagneticcomparability(EMC),surgewithstandcapability($WC),and software etsign/V&Y. We also noted that the licensee had not addressed these areas in cetail in PDCR 861.
In light of these shortcomings of the higher level cesign documents, the staff tvaluated lower level design documentation and audited both the vendor's and licensee's design process to assess the degree of conformance to the above crittria.
RE3 GUIDE 1.152, AN51/IEEE ANS-7-4.3.2-1982 conformance:
Regarding conformance with ANS!/IEEE ANS 7 4.3.2-1982, the licensee had directed the vendor to develop a statement of confonnance to this standard for the vendor's design and design process. This standard primarily identifies application criteria for :omputer system design to be addressed and documented in the design, rather than specific functional criteria.
This standard was developed to amplify lEEE Std. 603 1980 because of the unique nature of digital computer systems (sof tware). Functional criteria and cesign basis requirements are more completely adcressed by !EEE Std 603 and its
- daughter" criteria (such asIEEEStd379).
During the visit to the vendor's facility the staff confirmed that the vendor denonstrated conformance to the majo,r elements of AN$1/IEEE ANS-7-4.3.2-1982. This is further discussed in section 5.2 of this report, wherein some shortcomings in implementation are identified.
?
!EEE Std 003 1980 conformance:
IEEE Std 603 1980 did not appear in the licensee's system specification.
although it 15 stipulated in the PDCR 861 and is addressed in the vendor's conf omance document.
In sumary, the vendor (Foxboro) has claimed conformance capability, and repair provisions of IEEE Std 603.to the quality, equi isolat "he l as system integrator) has claimed conformance to these and the remaining technical provisions of IEEE Std 603.
These remaining provisions include: establishment of safety system design basis; single failure criterion completion of protective action; independence; information displays; a;uxiliary features; ano i
functional / design requirements dor sense, comand, and execute features.
On the basis of the sample failure modes investigated and on the basis of the RPS coincidence logic architecture which has been preserved from the original design failure, the review team concluced that the upgrade appears to nett the s' ogle criterion. However, as discussed in Section 5.1.2 of this SER, the 1
i staff noted that no FMEA had been performec for the new system that would demonstrate conformance more conclusively.
The licensee was asked to determine t
if an FHEA was required as a part of the HNP design basis. The licensee respor4ed in their May 5 either the original or re, placement system.1987 letter that a detailed TMEA was not p However, an analysis of the system was ccepleted and cocumented in the FSAR section 7.2.2.
This analysis did a comparison of new equipment failure probabilities to those of old equipment in accordance with IEEE 577, ' Standard Requirecents for Reliability Analysis in i
the Design and Operation of Safety Systems for Nuclear Power Generating Stations."
The results showed that the upgrade would likely enhance the reliability of the system and thus have a potential positive effect on plant safety and availability.
Regarding completion of protective action the licensee was requested to confirmthatresetofthewatchdogtimerfollowingadefaulttripinitiatedby the timer would not unistch the reactor trip output signal.
In the letter of i
i b y 5, 1989 the licensee confirmed that an initiated trip signal will not be l
automatically reset. Following a watchdog tineout the card will go to the FAIL If the error which caused the timeout was transient the card will try to mcce.
reset every 250ms and if successful will reset to the ERROR STANDBY mode which Leeps the output contacts in the same state as FAIL. Positive action must be taken by the operator to completely reset the caro to normal operation.
Regarding conformance to independence requirements, the licensee demonstrated that protection / control interfaces are hardwired through isolators, and no cigital data links are employed; thus, there is no fundamental change in the design features provided for control / protection independence. Physical separation and isolation are discussed under the topic of IEEE Std 384 conformance.
Regarding capability for test and calibration, the licensee discussed the test procedures and indireted that the upgraded modules were transparent to the RPS.
No credit is taken for the additional on-line diagnostics provided with the new 1
I
. - _ ~ _
__ = -
O o
l l
i modules although the diagnostics will provide additional maintenance tools.
ThestaffagreesthattheupgradedPodulesasintegratedintotheRPSinPhase I meets the intent of this provision of IEEE 5td 603.
Based on the preceding degree of conformance claimed by the licensee and the staff aucit of the design documentation (discussed further in Section 5.1.1.2 TunctionalRequirements),thestaffconcludesthattheupgradeconformstoIElE l
Std 603.
l IEEE Std 384/RG 1.75 conformance:
These requirements were not a part of the original HNP design basis, but the staff believes that new equipment should be procured to these requirements and l
that the installation should conform to these requirements to the extent practical; in no cases should the modification reduce the degree of inhpendence, separation, and isolation provided in the original Wesign.
l The licensee's system specification stipulates RG 1.75 1974, and the PDCR 861 stipulates RG 1.75 1978 and IEEE Std 384 1981, secs 7.2.1 and 7.2.2 (isolation I
of instrunentation and control circuits).
Exception is taken by the licensee l
in PDCR 861 to IEEE Std 3841981 Sec. 6.6.2, in that no electrical or physical isolation is planned between sections of individual channels in some sections l
Of the control room. The upgrade will not bring all aspects of the RPS at j
haddam Neck into conformance with these criteria. This is not a change from the intent of the original licensing basis and therefore the staff find this acceptable.
The Foxboro L2CR output isolator module used in this design was tested for n.eximum credible fault of 600 vac via a transformer with an output rating of i
630 amps. The device die not allow the fault to propagate to tie Class LE side and is therefore acceptable to the staff.
Ee arding field interf aces to the upgraded system, the licensee stated that PD R 861 resulted in separating the analog portions of the existing RPS into four input channels to the RPS relay coincidence logic, and that separation of the inputs to the RPS logic has been improved relat've to the ori i
The licensee reported that two racks with barriers were provided <ginal design, or the r
upgrade to provide separation of RPS inouts and control / protection circuits.
Tour separate raceways are provided connecting the four channels from the two racks to the control board, and protection / control separation within each raceway is achieved by vertical barriers within each tray. The staff concludes that this appears to be a significant improvement in physical separation / isolation over the original design, and is therefore acceptable.
lEEE Std 323-1974 conformance:
I IEEE Std 323 1974 was properly stipulated in the Itcensee's system specification, PDCR 861, and in the vendor hardware specifications. The licensee stated that an environmental qualification (EQ) review was performed that included a verification of the final installation arrangement of Class 1E
o b
e
\\
i i
equipment using EQ walkdown checklists.
i The upgraded Spec 200 Picro instrumentation is located in a mild environment, but t te review team' required assurance that localised high temperatures in the rack were within accept 6ble limits under design basis conditions, i
i The staff reviewed test reports (00AAB69 Rev. A) demonstrating temperature rise l
at various locations within a representative cabinet. Acceptance criteria for the tests were that no more than 0.5 a change in ambient temperature of 28gercent shift in accuracy should occur for C within the limits of 5'C to 50'C. The tests were reported as successful, and the equipment appears to be qualified for the application on that basis. However the Foxboro Spec 200 Micro User Notes indicates that if more than three con, trol cards are installed in a nest, t
the heat rise can be very high and may affect long term reliabilitf. This i
vendor recomendation note was a result of the thermal / life testing of the The staff asked the licensee to determine if they had 'nstalled equipment.
more than three control cards in a nest, and they detetmined from the i
conf t uration marval that in Rack DR, Nest 3 four control cards had been i
instaled,exceedingthevendorrecommendedlimit. The staff believes that the i
licensee should adhere to the vendor recommendation or adequately justify any exceptions.
In the May 5,1989 submittal the Itcensee reported that the rack with 4 nested control cards was used only for non safety.related control and therefore coes not require staff approval. The staff agrees.
i IEEE Std 344 1975 conformance:
1EEE Std 344 1975 was properly stipulated in the licensee's system t
specification, PDCR 861, and in the vendor hardware specifications.
The licensee stated that a seismic qualification review was conducted, including veritication of the seismic integrity of Class 1E equipment and equipment mounting. Detailed review of the licensee's seismic analysis was not in the l
scope of this evaluation; however, we noted that the a seismic qualification test report (Q0AAA20-1) showed that the Spec 200 Micro hardware in i
representative rack configurations, remained functional during and following thevendor'ssafeshutdownearthquake(S$t)excitationexceptfortheFoxboro L2CR output relay isolator which In the IIcensee' experienced unacceptable contact chatter during the test.
s application, these devices are critical because they are used as interfaces to the RPS trip logic. These devices were i
1 successfully tested to the vendor's operating basis earthquake (OBE) levels, but not to the vendor's $$E levels.
l In the May 5, 1989 submittal the licensee reported that the lower level vendor OBE test which had no equipment malfunctions did envelope both the OBE and $5E spectra for Haddam Neck Plant and is therefore acceptable to the staff.
5.1.1.2 Functional Requirements Detailed functional requirements for the RPS were provided by the N$$$ vendor specification, PDCR 861, and the licensee's system specification.
o 5
14 The licensee provided supporting docunwntation and a discussion regarding the developsent of the functional requirements.
The licenste had prov'det functional block diagrams and other documentation of the RP1 as it entsted before and after the upgrace.
To assess the implerentation of design basis functional requirerents in the upgraded RPS, the NRC review team perfomed an audit of the low pressure scram function.
The audit used the " thread concept" consisting of the variable low pressure scram input channels, scram calculators, and output channels to the existing RPS coincidence trip logic, A variable low pressure trip signal is penerated whenever the pressuriter i
pressure is lower than either the calcu ated low pressure setpoint or the fixed lower limit.
The calev1sted setpoint is a function of the reactor core hot channel outitt temperature (Theo). The Tave and eelta T signals are transmitted to a calculator which con hot channel factor, core flow bypass &utes Thee. Thco is calculated from Tavg.
fraction gain constants, and transfer functions that compensate for instrument delay,s and transport / mixing delays between the core ano the steam generator. Each of the three indepensent low pressure scram calculation outputs provides an input to a low pressure scram comparator, a low pressure serem pressure indicator, and a recorder. Each low pressure scram comparator also receives inputs from one of three independent pressurizer pressure channels.
Each comparator provides an alarm and a scram signal to the existing low pressure scram coincidence logic when the measured system pressure is at or below the calculated scram pressure. Any trip signal generated from these channels is transmittee to matrix of relay contacts which initiates a reactor trip upon coincidence of two out of three signals.
Thestaffreviewedthisscramfunction(thread)withthelicensee using their system specification and functional diagrams. The functional requIrerents in these documents were compared with the original design basis and the licensee's configuration manual for the Spec 200 Micro system. The licensee walked through a line by.line comparison of the configuration, comparing it with the system (procurement) specification and the functional diagrams. The staff noted that some changes had occurred relative to the original functional requirements whereby some constants and transfer functions in the configured system did not agree with the original design basis. The licensee was aware of these differences, and provided two letters from the NS$$ vendor from 1967 and 1971. These letters resolved the differences; the 1967 letter changed lead / leg constants to reduce the effects of noise on the calculated setpointst the 1971 letter revised the setpoint equation, and changes were also made to account for fast response RTDs. Taouth the specifications should have reflected the latest requirements the requireunts were properly implerented.
The Spec 200 Micro cards have been designed so that the output contacts can be configured by the user to either close or open the output contact upon module failure (loss of power).
The plant documentation showed that the Spec 200 Micro cards were correctly,fumpered for the required failure modes for the thread examined.
The licensee demonstrated that response tires required by the UFSAR are achievable with acceptable margin by the upgraded RPS channels, ano that the
s
- 15 tynamic response of the upgraded 61gital system are functionally equivalent to the analog system.
The licensee walked through test results to demonstrate that the digital system's response to a step input was recorded and then compared to the calculated response. The system performed as expected. On that basis, the review team believes that this provides a reasonable assurance that the systems are dynamically equivalent for this situation.
5.1.1.3 Adequacy of Licensee's safety Analysis The licensee was asked to provide the safety analysis supporting the instrumentation and control system aspects of the design modification uncertaken in the upgrade of the RFS. This analysis should demonstrate that the mocification was in accorcance with the current plant design basis and supportable revisions to the design basis should be provided if necessary.
In 1984 Foxboro prepared a ' Reliability (Availability) Analysis of a Sotler Control System implemented with Spec 200 Micro and Spec 200 for comparison of Spec 200 Hiero and Spec 200.'
In developing the comparison the assumption was made that any failure of equipment would result in a total system failure and therefore should provide additional conservatism above the calculated results.
The component failure rates used were obtained from military reliability prediction and component stress analysis. A significant influence on the calculations was the amount of equipment required to implement the control system.
The Spec 200 Micro system has one microprocessor card per parameter that performs all r.ath and control functions whiie the analog Spec 200 uses separate modules to perform each function. A single analog card will be more reliable than the more complex microprocessor card. However, the sample control system implemented with the Spec 200 micro was estimated to be 255 more reliable due to the use of one microprocessor module instead of several analog mooules.
The licensee stated that they received only one vendor product safety notification, which the vendor uses to inform their customers that a component which they provided may have a problem. This notification applied to Spec 200 Micro equipment that was procured as non-QA Category 1 for the training department, and apparently involved the failure of a control card. The work orcer which replaced the failed card identified the failure as a watchdog timer timeout. The licensee and vendor were unable to retrieve documentation resolving the failure since the procurement was non safety related.
The licensee also performed a Reliability Analysis in 1987 which addressed the reliability of the modernization of the Haddam Neck Plant Reactor Protection and Control System as described in POCR 861. The licensee identified that the existing Taylur recorder / controllers and original Foxboro equipment were about 20 years old and recent inspections have shown widespread instances of age degradation. Some equipnent used at Haddam Neck Plant has been out of production since 1968 and spare parts have been increasingly difficult to obtain. The licensee concluded that modernization of the reactor protection and control systems should result in a reduction in the amount of time spent
o*
}.
r for equipment repair and maintenance, enhancement of equipeent reliability, and a potential positive availability gain.
5.1.2 Failure Modes and Effects 5.1.2.1 Detectability of Failures i
Detectability of failures in a safety system is critical when considering failure moces, since all nondetectable failures identified in the failure analysis must be assumed to occur. Failures in a system ma system or channel level by periodic testing, surveillance,y be detected at the alarms, or ciagnostics.
IEEE Sto 379 provides specific guidance for application of the Single Failure Criterion to a safet) system and detectability of failures.
This stancard was invoked by the licenses for the new system, i
The licensee stated that on-line diagnostics are provided with the system to continually monitor system status ano pruvide system alarms upon failure.
Local indications are provided to facilitate identification of a failed cardt inv411e(out-of-range)inputsignalscanbedetected,andvalidationofinputs t
is provided. A hardware watchdog timer is provided to detect system stall, and l
time out of this timer will place the card in a trip mode. Though the new equipment provides self diagnostics the licensee has stated that they will use this feature for maintenance only and they will maintain the same plant on-line I
testing of the RPS as before. This is acceptable to the staff.
j 5.1.2.2 Special failure Modes 4
IEEE 603 1980 requires in part that "... safety systems shall be designed to i
accomplish their safety functions under the full range of appitcable conditions 4
enuterated ir. the design basis."
This section presents consideration of f ailure modes that miqht be peculiar to a digital system, such as system stall and timing errors.
, n sedition, some fundamental design basis failure modes are discussed.
The Spec 200 Micro design incorporates a hardware watchdog timer which can detect a system stall.
A time-out of this timer will place the card in a trip mode.
In the absence of an FMEA. the staff requested that the licensee provide the results of a power-down and recovery test on the u effects of removal / reinsertion of a circuit card. pgraded RPS, and assess the j
For the power interruption scenario, the licensee retrieved test results demonstrating that the output of the installed system modules following remuval of power were as configured; however, the licensee did not extend this test to include immediate power restoration. The licensee stated that the correct status of the modules following restoration of power was confirmed at the next t
system surveillance fullowing power-up.
For the card removal / reinsertion licensee stated that this scenario was bounced by the worst-case failures, the l
o 3
i s
1 assumed for the RPS (as configured) and by the power down test.
This is acceptable.
While the foregoing provides some assurance that new failure modes and effects have not been introduced, the review team determined that no formal FMEA was implemented. Noting that a digital system may have different failure modes than an analog system (such as system stall or timing errors), the review team believes it would have been prudent to perforin a comprehensive FMEA, to confirm that the original design basis is maintained.
However since the system is essentially a one for one replacement of the analog por, tion of the RPS (and not the logic), is equipped with hardware watchdog timers, and the output contact i
i state is cetermined for the most likely failures, absence of an FMEA is accept.able for Phase 1 of the upgrade.
5.1.2.3 Acceptance Testing i
The licensee provided the test plans, test procedures, acceptance test criteria, and test results which demonstrate reasonable assurance that the installed system functions as required by the plant design basis. The licensee reported that site acceptance of the equipment associated with the RPS upgrade was conducted in various phases under a hierarchy of procedures including bench calibrations / response time testing of transmitters;f relay); c chant.el functional tests (from transmitter to actuation o channel calibrationst
)
response time tests; and logic matrix checks.
Based on our audit, the staff i
finds the acceptance testing program acceptable.
5.1.3 Effects of Retrofit of New Equipment to Existing Systems This modification incorporates equipment which has been designed to current i
technology requiremerts into a safety system with a design based upon a previous technology. These digital based processors must be installed in a manner which will protect against the comparatively high current and voltage sigtels present in the analog based designs.
Signal isolation devices are utilized in the Haddam Neck upgraded system to perform a portion of this isolation / separation. There remains however, the radio frequency radiation and conductance interferenc,e paths which typically occur a(RF) t frequencies which are beyond the concern of the analog designs but which may adversely affect the digital equipment.
Currently no specific surge withstand capability ($WC) electromagnetic interference (EMI) standards or criteria are firmly established for designs utilizing microprocessors at nuclear plants.
commercial and military standards (MIL STD 461Some utilities have used existing MIL-STD 462 and SAMA STD PM (33.1 1978), " Electromagnetic Susceptibility of Process Control Instrumentation") to prepare test procedures.
Since the microprocessor technology did not exist during the period of the original plant design and esta11ation the existing plant criteria do not offer guidance for the protection of the new design from these effects. The introduction of interfering or damaging energy levels into the new equipment which is installed 1
L 9
\\
y within existing equipment racks and powered from existing power sourcet needs to be considered in the retrofit design. EMI failure modes have occurred at other plants with different vendors solid state modules. For example during preoperational testing at the Hope Creek Generating Station in 1984, Incorrect actuation of Bailey $62 solid state logic modules were caused by voltages (EMI) being induced in the non energized conductors ey energized conductors in the same or adjacent cable. Over 50 licensee event reports describing EMI related events have been documented from various plants utilizing solid state modules.
Because of these events and similar problems at non nuclear applications the staff requests that solid state and microprocessor equipment be evaluated for the installed environment, including [HI and SWC.
Evidence of the consideratiun of the application of SWC and EMI standards or the establishment of design control limits was requested of the licensee. The review of system tests to provide assurance that the operational characteristics of the equipment as installed within the plant systems, including the power connections and system grounding, was also requested from the licensee.
No such considerations were initially retrievable by the licensee, except for a "walkie talkie" test of limited scope and usefulness.
The NRC issued Information Notice 83 83, "Use of Portable Radio Transmitters Insice Nuclear Power Plants" on December 19, 1983. While the licensees' "Walkie Talkie" test addressed the specific cases described in the Inform,ation Notice, the notice notes that when solid state equipment is retrofitted into en existing plant, the potential for RFI vulnerability suggests that the licensee should evaluate the impact on plant operation and safety.
Possible sources of RFI other than portable radios should be addressed.
Discussions with the vendor did indicate that in addition to their normal proeuct development testing a test program conducted for a foreign utility in response to the utility's test requirements had been performed.
The staff noted that this testing was performeo and the report issued in late 1988, more than 6 months after the HNP RPS upgrade was operational. The tests included electrostatic discher e, conmon mode rejection nornal mode rejection, with frequency transient e fects SWC, lightnin of ects and EMI. ANS!/IEEE,IEC andvendorstandard,andcrIteriawereutiited.
In addition to the above tests, the vendor also cited tests evaluating the effects of conducted and radiated emissions for a Spec 200 Micro configuration emp oying a network communications module in addition to the control cards and aux liary modules tested for a stand-alone configuration.
Emissions from the new equipment was also evaluated. All of the tests were reported es successful, The staff finds that the SWC/EMI testing is acceptable and has established a level of SWC/EMI qualification.
5.1.3.1 Design Basis Surge Levels on Power Connections & EMI The RPS modules are powered from four separate class IE instrument buses, which are powered from a station battery backed DC power bus via DC/AC inverters. The specific inverters used at hhp produce a square wave voltage which is a source
O l
\\
l
{ l of broadband frequencies of potential concern to sensitive instrumentation.
- A hRC Office for Analysis and Evaluation of Operational Data Case Study Report, (At00/C605 December 1986)*, described many cases of operational experience i
involvingIossesofelectricalinverters.
Inverters of the vintage used at HNP j
appear to generate more power line transients than the new designs. The staff believes that installation of solid state and microprocessor equipment in systems which are powered by relatively (electrically) noisy inverters require particular attention.
The $pec 200 micro equipment must be shown to operate properly when powered from these sources. The vendor's test program has established a level of qualification, but the licensee was unable to retrieve any reprasentative measurements of the instrument bus waveforms and transient characteristics, to assure that the electrical environnent experienced by the RPS is enveloped by l
the vendor's qualification. The licensee needs to determine what the speciftc i
operating environment is.
Consequently the licensee should implement a program to monitor instrument bus power, characteristics (including surge, total harmonic distortion, and transient impulse) to assure the installation is qualified.
t The monitoring program should be conducted ovtr a sufficient period of time to provide representative measurements of switching transients.
i I
One method which was acceptable to the staff was described in
- NURIG/CR 3270 Investigation of Electromagnetic Interference (EMI) Levels in Commercial,
4
^
NuclearPowerplants,(August 1983).*
The licensee should use this or some other acceptable method (such as milspecs) to detemine that the vendor qualification testing envelopes the installed configuratten.
Based on the existence of extensive vendor qualification of the RPS upgrade modules, and the t
comparatively limited scope of the RPS upgrade to date, the staff concludes that short term operation of the upgraded RFS should be acceptable, but the qualification of the system to the electrical environment remains unresolved until representative data is available to confirm that the bus power i
characteristics at Haddara Neck Plant have been enveloped by the vendor qualification program.
Information regarding the IH1 levels in the imediate vicinity of the cabinets which house the RP$ units was not available. The licensee prohibits the use of portable radio transmitters in the Control Room where the eovipnent is located.
The issue of interest here is possible susceptance of the new digital eculpment 1
to a corrnon mode threat from high frequency radiation from signa s in atjacent cables or by grounding system interconnection to other signal sources within the plant; forthesesiteatiers,adesignanalysisortesting(NUREG/CR3270) should be performed which sodresses these limits and possible effects.
If analysis is used in 11eu of testing, then in addition to the normal plant electrical documentation, grounding descriptions should be performed to define l
the ground paths at high frequencies in addition to the normal low frequency path descriptions.
The vendor has provided a degree of qualification of the hardware supplied for the upgrade, but the licensee has not shown this to
. envelope the plant environment. The licensee has concluded that the low EMI susceptibility demonstrated by Foxboro bounds the Haddam Neck installation.
4 Y
m i
,.5
\\
r i ;
4 The staff does not accept the licensee conclusion. The licensee should perform the Ett1/SWC analysis or test described above and submit the results. Because Fhase !! will add substantially more new microprocessor equipment, the licensee is required to submit a program plan describirq the analysis, testin schedule to resolve this concern prior to restart following Phase !!g and implementation.
I S.1.3.2 Suitability 01' Existing Cables for New Instrumentation The installation of the new equipment has been accomplished by the utilization of existing equipment, cabinets and cabling as appropriate.
The existing configuration of the RPS provides some design features such as shielded /twistec pair analog input cabling with shields grounded at a single point. Also physical separation of channel steal and power inputs using conduit and l
barriers /enclocure in much of '.14 installation. This method is acceptable to l-the staff.
5.2 SOFTWARE ASSESSMENT The focus of our audit of the Spec 200 Micro software assessment is on the methodology and procedures used to develop the modules. Assessment of the sof tware development methodology was done by reviewing the verification and validation through the development process.
Verification and validation (V&V) are two separate but related activities that follow the development of software. Verification determinas whether the requ!rements of one phase of the development cycle have been consistently, correctly, and completely tran:; formed to the subsequent phase of the cycle.
Va11 cation is the testing of the final product to ensure that performance of the end product conforms to the functional, performance and interface requiremer,ts of the initial specification. The need for V8V arose because sof tware is very labor intensive, and prone to human errors of omission, commission and interpretation.
V&V provides for a reviewer to work in parallel with, but independent of, the development team to ensure that human errors do not hinder the production of quality software that is reliable and testable.
In executing V8V, certain principles have proven over time to be very effective in software development programs. These verification and validation principles can serve as a comprehensive reference base for applying the applicable, criteria for sof tware evaluations of Class 1E safety systems.
I') Well defined Systems Requirements expressed in a comprehensive document.
- 2) Development Methodology to guide the production of software.
- 3) Comprehensive Testing procedures.
- 4) Independence of the V&Y reviewer from the development organization.
A
4;
's i
\\
. 1' 5.2.1 Criteria t
The applicable criteria for the developnent of safety related software products l
is set forth in ANSI /IEEE-ANS-7-4.3.2 as endorsed by RG 1.152. This standard i
defines the documentation of the computer system requirements, the software development phases, the verification testing of the integrated system, and the validatiun of the entire resultant system. This audit focuses on whether a ce,mprehensive Y&Y program has been applied to the software development of safety related systems, and that the V8V program was carried out according to the applicable criteria. The V&V provides tlie traceability necessary to determine whether the criteria for safety software systems are satisfied. Using this structured methodology provides the vendor licensee and the staff with a high confidence level that tie end product will, perform as required, i
l 5.2.1.1 Verification.
The verification process traces the development of the safety software through the phases of requirements design and implementation. At each stage of the process,theverifiershouldreportwhethertherequirementshavebeen l
successfully translated from the previous phase to the current phase. Since the purpose of verification is to ensure that requirements have been transformed correctly, the verification process should be carried out explicitly at the functional requirements level.
5.2.1.2 Indeper.dence.
A key element in effective verification is that the verifier must be incependent of the developing organization. Although the verifier must work closely with the develo> ment team throughout the software life-cycle, he should not be connected with t.e development project.
In this way there is reasonable assurance that the verifier is insulated from the cost and schedule pressures of the development team and can exercise independent judgement. The staff continues to interpert IEEE/ANS 7-4.3-2 to require that the independent verifier must, as a minimum, answer to a different fiist line manager.
5.2.1.3 Testing.
Integration testing done during the develo) ment of individual modules and integration of these modules must address )oth the static and dynamic behavior of the software with a comprehensive application of test data. The static testing should determine whether the module or integration of modules are behaving correctly for given inputs and outputs. The dynamic integration testing should simulate the actual conditions that the suftware must perform under, in particular with regard to timing, and transient conditions like half trips, and power interruptions. The execution of these tests must be documented in reports that include test results and errors uncovered.
Verification testing must demonstrate that individual functions implemented by each module perform as required by the design specification. The test must cover all of the functional requirements to ensures that all aspects of the
t module or subsystem have been tested. Verification testir,g is proceeded by detailed test plans and procedures that specify the test data to be applied and the test environnent, including the methodology for capturing the results for analysis and reporting.
The test plan should be comprehensive and provide sufficient detail to be the basis for evaluating the test coverage.
1 5.2.1.4 Validation Validation is the test and evaluation of the integrated computer system to ensure that it complies with the functional and performance requirements of the system specification. There shall be a forwal validation test plan that identifies the tests for function specifying inputs and expected outputs. The validation test plan shall be exec,uted and the results shall be retained in a form that is auditable by persens not connected with test program. The test i
report shall summarize the test results and show how the system is in compliance with the requirements. Validation Testing must be done by a team that did not participate in the design or im product and not the same first line manager.plementation*of the software 5.2.1.5 Discrepancy Resolution.
A key element in any Verification and Validation effort is the process by which
~
discrepancies uncovered during Verification are recorded, flected in all identified, resolved anc corrected. The resolution of a discrepancy must be re applicable documents, whether source code, the software design specification, the software requirements, or the original systems functional requirements.
Formal discrepancy resolution with specific sign-offs imparts a degree of configuration control on the software product ano the accompanying documentation.
5.2.1.6 Configuration Management.
Software is flexible, changeable and often ambiguous. Configuration management procedures ensure that the development team as well as the verifier know exactly what is being built and tested. As discrepancies are corrected, and mocules modified, it must be clearly denoted which are the new versions of the modules and how they differ from the old versions. Change control and the response to discrepancies uncovered during verification and integration testing is important to traceable software development. As the software design evolves, and the software implementation is passed from one group to another, it must be under configuration control, so that any subsequent changes are made to a clearly defined baseline.
5.2.2 V&V Plan The V&V plan for the Spec 200 Micro is represented by Foxboro document no.
00AAE03 - Rev 8 dated 26 October 1988. There was no attempt to mimic IEEE 7-4.3.2indevelopingtheYavPlan,butrathertodescribetheorderly development process that was followed for Spec 200 Micro software. A comparison was provided to ANSI /IEEE 730-1984 " Software Quality Assurance".
1 s
'I g
,a
= 23 -
i i
t l
The sof tware development plan was combined with the verification plan. The conformance of the Spec 200 Micro sof tware development to IEEE 7-4.3.2 is cescribed in Foxboro document Q0AAE02, Rev. A, dated 14 March 1988. In general l
all of the important elements are present, although it was difficult to follow because many of the requirements are satisfied in other documents that are included by reference. The references shoulo be more specific in instances where specific requirements are being addressed.
1) 00AAE02 provides a step by step comparison of the Foxboro V8V Plan to IEEE 7-4.3.2. Section 3.1 Hardware Requirement of IEEE 7-4.3.2. requires that the hardware requirements which impact software be specified and verified and provides a specific list of eight events which must be specified as a minimum. Each of the required areas was addressed by Q0AAE02*and the i
referenced cocuments. The staff found each of the eight sections acceptable.
Section 3.2 " Software Requirements" of IEEE 7-4.3.2 provides a list of 12 minimum sof tware requirements which must be s pecified.
Each of the e
required 12 areas was addressed specifically by Q0AAE02 and the referenced l
documents. The staff found each of the 12 sections acceptable.
2)
Section 3.3 " Hardware-Sof tware Integ&V plan was first developed in 1 ration Requirements" of IEEE 7-4.3.2 was not satisfied by Q0AAE02. The V March 1988, after the Spec 200 Micro software was first integrated as Version T1 in June 1985. This is a month after the most recent version (SA-12) was released to the field on 16 February 1988. Therefore, the V8V plan was not available for use during the early stages of software development. Though Sof tware Quality Assurance (IEEE 730) was followed and most of the requirements were satisfied the staff finds that this document did not fully satisfy the criteria of IEEE 7-4.3.2.
3)
Section 4 of IEEE 7-4.3.2 addresses the software development plan, design and implementation. The staff found that most of the requirements were met except that the plan lists the names of individuals who were or are responsible for various aspects of the Spec 200 Nicro software, but it did not clearly define the organizational relationship between these people.
The documentation structure for the Spec 200 Micro was well defined with explanations in cases where design or production changes resulted in the merger of two documents. The inclusion cf the Detailed Functional Specification in the Corporate Product Specification was specifically mentioned. Additional specifics of how the V&V plan was implemented are provided in the following sections.
The software development plan described the organizational and procedural aspects of the development of the Spec 200 Micro software. These included the definition of the development phases, and topical descriptions of the documents retlecting the results of each phase. Technical reviews that were performed were identified, but only described in a general manner. The interaction and the respective respcnsibilities of the cevelopment and the quality assurance
a s
- 24 i
teams were also delineated but not clearly with regard to reporting. The review team concluded the document was well written with regard to the i
develcpment of the software and complied with the general intent of AN$1/IEEE-AliS-7-4.3.2 not address the specifics of requirements traceabilityalthough structura and personal responsibility as expressed by sign-offs., verifier independence, 5.2.3 Design Process
[
The specification and the subsequent design of the safety system was reviewed to identify the consequences of failure. The design of the Spec 200 Micro
?
software began with the development of the Program Technical Description (PTD) for rech Control Block from the functional specifications of related analog control cards. There was no evidence of a specific methodology or analysis used to develop and refine the analog functional requirements into software PTD's.
The design process is described in Section 4.3.3.3 of Foxboro Q0AAEE03 (V&Y) and includes walkthroughs of the pseudo code as a review methodology.
The walkthrough for cal.C was done by 6 people on March 15,1985. All of the people involved in the walkthrough were from the development group.
While the design process exhibited thought and evaluation and demonstrated conformance with most of Section 4.2 of IEEE 7-4.3.2, it was not evident how the individual analog requirements were decomposed into the more detailed requirements of each Control Block in a treceable manner.
Because most of the requirements have been met and validation testing showed that the digital implementation accurately duplicated the analog design, the staff concludes that the the design process was acceptable.
5.2.4 Implementation Process IEEE 7-4.3.2 Section 4.3 requires that che implementation procedures be developed ano documented.
The process of software development for safety related systems has to be methodical and controlled so that the status of the sof tware is known at any time. A well defined and controlled development I
processprovidesagoodenvironmentforeffectIveverificationaswellas l
management oversight. The development methodology for the Spec 200 Micro was described and provided an overall view of the major development phases and quality assurance activities that appeared to be methodical and controlled.
The major tool in the verification of software is the walkthrough of algorithms and the deskehecking of code. This is especially true of the Spec 200 Micro with its extensive algorithms, data vectors, and commission logic.
The Implementation Phase was described in the ANSI /IEEE 7-4.3.2 conformance manual and stressed walkthroughs as the primary means of review. The existing documentation reports for the walkthroughs indicated that all the people involved were all from the development group responsible for the Spec 200 Micro.
l There did not seem to be any independent review of the implementation.
l l
l
o'
+
The walkthroughs were limited by company policy to two hours, which represents about 300 lines of code.
which has about 1200 lines of code, would require 4 walkthroughs.Th evidence of only one source code walkthrough for CALC.
There was The walkthrough for a part of the CALC source code was done 29 March 1985 barely two weeks after the CALC design was reviewed. Many of the reviewers, were the same and may of the errors uncovered we.s also the same. The staff 1
believes that, at least in this case, the walkthrough was not sufficiently independent and did not provide the additional verification that was intended.
it appeared that there was no error followup in the intervening two weeks.
No written evidence was found whether the errors uncovered in the design walkthrough were corrected at any point in the implementation process.
The walkthroughs took place before the initial release and much of the occumentation was informal.
The following questions were not documented.
1 Who corrected the error?
2 How was the correction made?
3 Who approved the correction?
4 Uhich version of the software corrected the error?
5 When was the error corrected?
6)
What document changes had to be made to reflect the error correction?
ANSI /IEEE-ANS-7-4.3.2 (Section 7.3) requires that the verifier interact with the design group in a written form. Such documentation provides traceability to the verification process procedures have been observe,d.and another level of assurance that proper During the meeting with the vendor, the NRC audit team was presented with no evidence of written interaction between the verifier ar.d the design team. The documentation consisted of the walkthrough reports, 60c a listing of the discovered problems. Following the audit, the vendor performed a review using independent verifiers not associated with the original design and documented that each error which had been found hao been corrected at that time oversight which has now, been corrected.and that this problem was limited to a documentat i
The staff concludes that the i.
implementation process is now acceptable.
5.2.5 Testing 5.2.5.1 Verification Testing.
The Verification Plan had specific procedures for veriftcation testing of the source coce including deskchecking and testing. From discussions with the vendor, it was indicated to the NRC audit team that the verifier had complete access to the program development activity. This included access to the source files for reviewing code, and the use of the development test bed for carrying out the verification tests. Much of the testing was done informally with no i
directly applicable test procedures, and no recording of test data. Although no written evidence exists, the staff believes that the verifier's access to
..-m
V*
. 4' 26 -
source code and to the development test bed did not prevent effective l
verification testing of the Spec 200 Micro.
5.2.5.2 Va11 cation plan.
Although much testing is cone at the module level by the individual groups responsible for building and developing the modules the integration of these i
modules is an important ifmiMons in the developme,nt process.
There are basically two types of testing during software integration; static and dynamic.
For the Spec 200 Micro members of the vendors g the static integration testing was carried out by development team using the emulator and target hardware. The tests were informal with test data developed as the test progressed. Each test was limited to addressing the specific behavior of the function in specific circumstances. There was no formal record of the integration testing.
i The dynamic integration testing was also performed by the vendor. The tes't procedures were described which referred to Foxboro Standards for dete41. A major tool used by Foxboro in conducting the tests was a series of software reliability models including the Duane Growth concept. These models measuredD u.
the goodness of the software and consisted of one or more of the following quantifiers:
Number of total errors, Number of remaining errors Program error density, l
Time estimate to next error, Rate of change of erro,rs, Error discovery l
profile, and Software defect profile The Duane Growth concept not only yields an estimate of the rate of change of errors, but estimates time to next error and number of errors to be expected in the near future. The Duane growth concept compares the detected error rate curing testing versus CPU operations. The staff audit team reviewed a company confidential sof tware reliability report, Report No. 87-SRR-001F, dated February 18, 1987. This report contained test data that was used in the Ouane growth concept for the SPEC 200 MICRO. We requested the licensee to documnnt the estimated time to the next high priority software error, which was docketed in the May 5, 1989 letter. A high priority error is one which will result in a loss of function. This estimate is based on the detected error rate during tests and the validity of the Duane growth concept. Our review of the report concluded that reasonable judgment was used in terminating the tests and releasing the SPEC 200 MICR0 for use. The staff also considered the operational history of the SPEC 200 MICRO, which consists of several hundred computer years of operation in the process industry. During this time, only one loss of function occurred in a control block module which is not used in i
the licensee's application. Based on the software reliability data, and the operational history of the SPEC 200 MICRO, we conclude the vendor had in place an adequate design and test program for the control blocks used by the licensee.
O
, ' g
r l,.
i 5.2.6 Configuration Management The staff concluded that the infonnality in testing and verification during the cevelopment of code was due to the apptrent lack of software configuration manegement procedures.
under configuration control as Version T1 in June 1986, after theThe S documentation, development and testing was completed. After this date there wassomedocumentationofthesubsequentcommercial(non-nuclear)relea,sesand the errors that were corrected.
procedures used for the versions.There was no evidence of the change control l
The licensee will be able to configure this equipment at the site and therefore the staff requested that the licensee demonstrate how they will control the configuration after installation.
trip function for a detailed evaluation.The NRC audit team selected the low pressure The staff requesten the licensee to walk thruugh the steps required to configure the micropror.essors for portions of this function.
The staff specifically requested the licensee to describe all operations upon the delta T (Thot - Tcold) signals in the low pressure trip :
function.
The walkthroughs of these signals began with a review of the i
transfer function's constants.
described the decomposition of the transfer fThe licensee produced documentation that block allocation within the microprocessors. unction in terms of the control The control blocks used were CALC, LLAG, and ALRM.
In the next step, the licensee produced the data sheets needed to configure each control block tu implement the transfer function.
In the audit process, each piece of data was evaluated and related to the desired transfer function.
From the walkthroughs, we noted the licensee's personnel were highly skilled in the use of the syntax needed to configure the niteroprocessor.
A test report was submitted by the licensee (May 5,1989) to demonstrate output signal response from the microprocessor implemented the transfer functfon to a
(
step input to the delta T signal.
Our review of the test report determined that the measured response of the output signal during the test accurately followed the calculated output ana was shown to be functionally equivalent to the previous analog system. Based on this data, the audit team concluded that the licensee successfully implemented the delta T signal within the low pressure trip function. Also, it appears reasonable to conclude that the licensee has successfully implemented the other functions performed in the upgraded protection system.
5.2.7 Software Conclusions The SPEC 200 Micro software development was initiated prior to the promulgation of RG 1.152 as the NRC star.dard for safety related software, for this reason it is understandable that some of the requirements of RG 1.152 were not
, strictly complied with. The key planning and controlling documents were written after the software development had been completed.
,s 28 -
l There was no indication of independent verification (as defined previously) of the steps in the development process. The docuirentation reviewed indicated that the vendor software reviews and testing was performed by people within the SPEC 200 micro development group.
The walkthroughs are an effective technique for designing and developing reliable seftware. Although the walkthroughs were used extensively on Spec 200 tiicro software, there was not much evidence that it was complete for the control blocks used in the licensee's system. The verification appears to have been effective as the followup validation testing has not shown significant problems.
The effectiveness of the walkthroughs was demonstrated by the error's that were uncovered. However, documentation of the correction of the uncovered errors l
was totally lacking. There was no way to determine whether the error was l
corrected, by whom, when, and its effect on the rest of the sof tware.
The walkthroughs were conducted entirely by the development group with no indepencent reviewers ptesent. Some independence was provided when new people to the project participated in the walkthrough. However their independence may have been compromised by the cost and schedule pressu,re of being in the l
development group.
The V&V Plan was produced after the software development was well underway.
This makes the plan more of a historical narrative rather than a plan for development. Furthermore the V&V Plan did not attempt to mimic IEEE 7-4.3.2 but rather to broadly cover the intent of IEEE 730-1984 " Software Quality Assurance Plans".
On June 20, 1989 the licensee submitted additional information which stated that independent walk-throughs were recently conducted on seven additional Foxboro SPEC 200 MICR0 configurable block types. These seven include:
- NONL - Non-linear Extender
- SWCH - Switch
- SSEL - Signal Selector
- RTIP - Ratio
- RAMP - Universal Ramp Generator
- ACUM - Accumulator
- INT - Integral Only Controller The walkthroughs were performed by individuals who had different first-line supervisors then the individuals who originally wrote the software. No code errors were found during these reviews.
In addition an independent review of all of the 21 block typts has been completed to verify that all software errors found and documented were corrected. The staff considers that this additional independent review provides additional assurance that the original software design is of high quality.
_ =
s9) 4 f
6.0 CONCLUSION
S 6.1 SYSTEMS AND HARDWARE ASSESSMENT CONCLUSIONS 6.1.1 General The NRC review team concludes the systems and hardware design of the RPS upgrade provides reasonable assur6nce that it will perform its safety functions as required by the UFSAR and Technical Specifications.
6.1.2.
Unresolved Electrical Environment of Installed System The staff requires, prior to restart from the Phase II RPS irnplementation, that the licensee submit a plan which documents the analysis, or testing which will be performed to demonstrate the electrical environment of the new equipment is enveloped by the vendors qualification testing.
6.2 Software Assessment Conclusions Although not in total strict conformance to ANSI /ANS IEEE-7-4.3.2 th development methodology was well structured and supported by ongoing,e comprehensive testing, particularly during system integration.
The validation testing and the extensive operational history of the software modules led to the overall conclusion that the vendor software can be considered acceptable for the RPS and that it can perform its safety function adequately as part of the RPS at the Haddam Neck Plact.
The staff believes that a few notes of caution are warranted. This SER addresses The Foxboro Spec 200 Micro equipment as it is a) plied at Haddam Neck.
1 The equipment has the capacity to be used for functions witch may be significantly more complex than evaluated in this SER.
It should not be assumed that this SER implies automatic NRC acceptance of very complicated (e.g. DNBR) system applications.
Use of the Foxaoro Spec 200 Micro in reactor flux trip circuits or other fast acting (under 200 msec) system applications is likewise not addressed by this SER.
It is reconnended to the Vendor that a sof tware development plan be written that can be applied to future Class IE software developments and modifications to the SPEC 200 Micro. The plan must be in conformance to ANS1/IEEE-ANS-7-4.3.2 and should include a description of the development i
phases in sufficient detail so that the verification and validation efforts can be initiated at the beginning of any design effort.
In particular the plan i
shuuld contain a taxono'ny of documentation reflecting development, procedures by which the reviews are conducted, procedures for ensuring the independence of verifiers, and a more ccmprehensive procedures to follow-up the correction of discrepancies.
Principal Contributor:
J. Stewart Dated:
March 21,1990 l
l 4
- -, - - -. - -