ML20059C930
| ML20059C930 | |
| Person / Time | |
|---|---|
| Site: | Cook, Zion File:ZionSolutions icon.png |
| Issue date: | 08/13/1993 |
| From: | Lewis H Advisory Committee on Reactor Safeguards |
| To: | Advisory Committee on Reactor Safeguards |
| References | |
| ACRS-2888, NUDOCS 9311020063 | |
| Download: ML20059C930 (13) | |
Text
.(
fERTIFIED BY:
DATE ISSUED: 8/10/93 9
Hal Lewis - 8/13/93
~
ADVISORY COMMITTEE ON REACTOR SAFEGUARDS COMPUTERS IN NUCLEAR POWER PLANT OPERATIONS h
SUBCOMMITTEE MEETING MINUTES:
GUIDELINES FOR ANALOG-TO-DIGITAL RETROFITS JULY 21, 1993 BETHESDA, MARYLAND INTRODUCTION The ACRS Subcommittee on Computers in Nuclear Power Plant Operations held a meeting on July 21, 1993, in Room P-110, 7920 Norfolk Avenue, Bethesda, Maryland to hear from the NRC staff and the Nuclear Utilities Management and Resource Council (NUMARC) on the progress of joint staff /NUMARC efforts to develop guidelines for the performance of 10 CFR 50.59 reviews on digital-instrumentation and control (I&C) system upgrades.
In addition, the Subcommittee heard from representatives of two utilities regarding their implementation of digital retro-fits for the replacement of analog instrumentation systems.
Finally, a brief presentation by the NRC Office of Regulatory Research (RES) was given on the status of digital system environmental qualification research.
The entire meeting was open to public attendance.
Mr.
D. Coe was the cognizant AORS staff engineer for this meeting. The presentation schedule for the meeting is attached.
The meeting was convened at 8:45 am and adjourned at 2:45 pm.
ATTENDEES ACRS H.
Lewis, Chairman T.
Kress, Member P.
Davis, Member J.
E.
Wilkins, Jr., Member W.
Kerr, Consultant P.
Place, Consultant D.
Coe, ACRS staff INDUSTRY T. Pietrangelo, NUMARC S.
- Brewer, D.C.
Cook Nuclear Plant R.
Mason, Zion Station W.
- Sotos, D.C.
Cook Nuclear Plant S.
Stimac, Zion Station FBC STAFF W.
Russell, NRR B.
Boger, NRR/DRCH J. Wermiel, NRR/DRCH/HICB A complete attendance list is included in the attachment.
There were no written comments or requests for time to make oral statements received from members of the public.
9311020063 930813 "r_
- - - ~
i Minut'es of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 CHAIRMAN'S OPENING REMARKS Dr. Lewis, the Subcommittee Chairman, convened the meeting at 8:45 am, and began by stating his view that a search for exact chresholds for any decision-making procedure will fail in the sense that precise criteria can never be found.
He noted that often there comes a point in such cases where ambiguous language is agreed upon simply to obtain agreement, recognizing that issues arising from this language must be dealt with later.
NRC STAFF PRESENTATION - Mr. Jared Wermiel Chief, I&C Branch, NRR Mr. Wenniel discussed the evolution of this issue since August 1992, when the staf f issued a proposed generic letter on analog-to-digital conversions for public comment.
This proposed generic letter sought to establish the principle that common-mode sof tware failure in redundant safety channels was a new failure mode which had not been previously analyzed or reviewed by the staf f, and thus was an unreviewed safety question (USQ) requiring staff review and approval before implementing the digital conversion.
Since the beginning of 1993, the staff has been working with NUMARC on the development of a guidance document which would contain review criteria for 10 CFR 50.59 evaluations, to allow for consistent determinations of digital upgrades which constitute an USQ.
The staff have reviewed the guidelines proposed by NUMARC and have returned them with comments.
The staff was primarily concerned that the draft NUMARC document did not define a clear enough
" threshold" for licensees to determine if a modification was an USQ.
In addition, the staff felt that training and qualification of operations and maintenance personnel must be reviewed when considering digital upgrades.
Mr. Wermiel stated that the staff would expect to reduce its review effort for those digital upgrades in which previous similar upgrades had already been reviewed and approved by the staff.
He further noted that the staff was unable to determine any means to answer the question of when staff review should be required, other than by " categorizing" the systems involved into those in which upgrades either must be reviewed by the staff, or may not necessarily require staff review.
NUMARC is evaluating these comments and expects to meet with the staff again in August.
Mr. Davis asked if the staf f believes sof tware errors must be shown to be impossible before deeming it unnecessary to design against the adverse effects of software failure.
Mr. Wermiel stated that the staf f has enough evidence and understanding of these systems to conclude that software cannot be made error-free.
Dr. Lewis asked if the staff had in mind a specific level of failure probability
-2 l
i
e
(
i 1
Minutes of ACRS Subcommittee on Computers in NPP Operations i
July 21, 1993 below which the staff would be unconcerned with software failure.
Mr. Wermiel said that the staff's concern is based on qualitative industry experience, and is sufficient to warrant closer staff attention to safety-related systems.
Dr.
Kerr asked if the staff has written guidance for its own reviews as well as those done by NRC contractors on licensee submitted digital upgrades.
Mr.
Russell indicated that such guidance was currently in the form of past staff safety evaluation reviews (SERs) for similar modifications.
He noted the staff is also continuing work on developing a standard review plan.
Dr. Lewis asked if, in lieu of concerns over the credibility of sof tware f ailures, the staf f had considered prescribing the review process as simply one in which knowledgeable people are required to review the proposed design.
Mr. Wermiel stated that the staff and industry are currently focusing on IEEE standard 7.4.3.2 which describes a software review process.
Mr. Russell added that the design acceptance criteria (DAC) for the standard plant design reviews are, in f act, a process being certified by the staff as an acceptable means to implement system performance specifications with digital control systems.
NUMARC PRESENTATION - Mr. Tony Pietrancelo. NUMARC Mr. Pietrangelo stated that NUMARC and EPRI had formed a joint committee in the Spring of 1992 to begin developing the guidance in their proposed draft now under discussion.
Approximately six to seven full-time EPRI staf f, plus contractor support, are dedicated to the entire area of digital control system design and licensing.
EPRI has also sponsored working groups in the sub-areas of electro-l magnetic interference (EMI) and software validation and verification (V&V) which include utility, vendor, and contractor participation.
The V&V working group includes contractors with expertise in formal V&V for software.
Mr. Pietrangelo further stated that the guidelines proposed by NUMARC/EPRI were modeled after the NUMARC/EPRI sponsored NSAC-125
" Guidelines for 10 CFR 50.59 Safety Evaluations," and sought to stay within the framework of 10 CFR 50.59.
This meant the outcome of the licensee's 10 CFR 50.59 evaluation should not be pre-determined as to whether a modification would result in an USQ.
Their draf t guidelines sought acknowledgement that certain digital upgrades being done at nuclear plants are extremely simple and, with good V&V, should not be an USQ.
They also included the probabilities of accidents as part of the probability for the failure of a digital system to perform its safety function.
-d t
Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 Finally, they viewed digital system failures only as initiators of the resultant " system-level" failure, and to evaluate them against similar " system level" failures previously analyzed and reviewed by the staff in the FSAR.
The staff's comments have indicated their preference for a more well-defined threshold, based on the type of system into which the digital modification will be installed.
In addition, NUMARC was
" disappointed at the [ staff's] discouraging tone" with regard to the use of commercial grade digital equipment.
Dr. Lewis pointed out that commercial microprocessors which have been manufactured and operated by the millions may provide better reliability than specially built systems.
NUMARC hopes to issue a revised document for public comment in September, resolve comments, and issue a final document by November 1993.
There is an expectation that the staff will endorse such a document.
Mr. Davis asked how much more effort is required of the licensee if it determines a digital upgrade is an USQ that requires staff review.
Mr. Pietrangelo answered the*,
although ideally there should be none, in actuality it requires more effort by the licensee to answer additional questions.
Mr. Davis noted that the staf f may also review any digital upgrades done under 10 CFR 50.59.
Dr. Kress asked if EMI was considered a serious safety concern.
Mr. Pietrangelo stated that EMI could be a concern if a digital upgrade were not installed properly, and that the EPRI: working group on EMI was trying to establish generic limits on the strength of EM.I signals which must be accounted for in the design and testing of a digital upgrade.
Dr. Lewis noted that the solutions to EMI are relatively easy to implement and should be general good i
practice for most electrical designs.
Mr. Russell offered to meet with the Committee in closed session to discuss some of the staf f's insights into EMI/RFI gained from discussions with the U.S.
military.
Dr. Lewis observed that the definition of " computer" in the NUl&RC draft guidelines referenced " programmable" digital computers, but noted that a computer did not really have to be programmable to be a computer.
Also, he stated that a staff letter he had read on this subject seemed to him to inappropriately mix up computers vs.
microprocessors, and software vs. hardware.
-4
4 i
L Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 LNDUSTRY EXPERIENCES WITH ANALOG-TO-DIGITAL CONVERSIONS Comments of Mr. Steve Brewer and Mr. Bill Sotos. D.C. Cook Nuclear Plant Mr. Brewer described the schedule for installation of the reactor protection system (RPS) digital upgrade currently in the final stages of NRC staff review.
Mr. Sotos described the details of the upgrade, which involved replacing Foxboro "H-line" analog equipment with comparable Foxboro " Spec 200" digital components.
He noted that the upgrade did not change any function of the RPS, did not alter any RPS or ESF
- logic, or change any existing field instruments or wiring.
He also noted that on-site personnel could not reprogram the modules, except for configuration changes which would fall under their Quality Assurance (QA) program.
In addition, the QA program monitors the performance of Foxboro by auditing documentation.
Mr. Sotos discussed the EMI testing performed to validate equipment susceptibility to EMI, noting that all acceptance criteria were
- met, although slight output shifts of one or two percent (comparable to analog equipment) were noted, Vendor recommended grounding practices prompted improvements to the cabinet grounding system.
He also stated that the plant has a very good overall grounding system and that lightning strikes have not caused equipment damage.
Dr.
Lewis noted that using the fact that something has not yet happened as an argument for not being concerned about it does not make sense, and that things which have the potential to do major damage must be considered.
Mr. Sotos stated that the 200 millisecond processor algorithm cycle time was considered during their review, and found not to require any changes to technical specifications for instrument response times.
Dr. Lewis noted that with a five megahertz clock speed, the algorithm requires one million clock cycles per execution cycle, and is thus a relatively complicated algorithm.
Mr. Sotos stated that their failure modes and effects analysis did not consider component level failures, instead using IEEE 352 as guidance for comparing the old system-level failure modes with those system-level f ailure modes associated with the new equipment.
The result of this analysis was that no new f ailure modes were identified for the new equipment.
In addition, IEEE 352 methods were used to compare the-reliabilities of the old vs.
new i
equipment, concluding that the newer equipment was more reliable.
Next, he noted that Foxboro used several statistical techniques to establish sof tware reliability, including Duane growth (error rate I
of change), error discovery rate, error Pareto profile, and McCabe 1
~.
a, l
\\;
Minutes of ACRS' Subcommittee on Computers in NPP Operations
' July 21, 1993 cyclomatic complexity analysis. The results of these analyses were found to be. acceptable, although Mr.
Sotos indicated that statistical methods yield "sof t" results compared t'o those achieved by experience derived from extensive commercial operating history.
Finally, he stated that because the diverse ATWS mitigation system (AMSAC) currently uses Foxboro " Spec 200" equipment, they will replace it with Taylor " Mod 30" equipment to maintain the required i
diversity from normal RPS.
Mr. Sotos stated that their biggest difficulties in the regulatory.
area were the lack of standards for-EMI and sof tware V&V.
In particular, current sof tware V&V standards such as ANSI-7.4.3.2-1982 do not address small non-user-accessible " stock" software as well as larger, custom-made, user-accessible software.
Dr. Kerr asked if they had found any problems during their review or testing which were unexpected or needed correcting.
Mr. Sotos explained that other than an unexpectedly high in-rush current, which required installation of additional limiting circuits, their review confirmed all their expectations.
He added that during testing at the vendor's facility they identified a generic relay failure problem, which was corrected by replacing all the defective relays.
Comments of Mr. Rick Mason, Zion Station Mr. Mason discussed the implementation of an Eagle 21 RPS upgrade at Zion which was reviewed and approved by the NRC staff.
The Eagle 21 system is based on the Westinghouse "7100" inicroprocessor, and replaces Foxboro "H-line" equipment.
The system retained the original RPS protection channel architecture, but compressed the number of required cabinets from sixteen to ten.
Expected benefits include better reliability over the lifetime of the plant, and the advantages of self-diagnostics and continuous self-calibration.
Mr. Mason noted that their original intent was to install the upg:.ide under 10 CFR 50.59, based on their evaluation that, f rom a overall system-level standpoint, the failure modes of the Eagle 21 system were the same as those of the older system.
However, the NRC staff reacted strongly to this position and subsequently the licensee decided to submit the project for staf f review. The staff review consisted, in
- part, of about 300 written questions requesting further information, and a five day technical audit.
The audit was performed by two NRC staff and three NRC contractors with each auditor concentrated on specific areas of review.
The areas of review were:
sof tware qualification (thread path audit of
]
one algorithm plus V&V plan reviews), hardware design and equipment
. i l
c 1
T.
Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 qualification, plant procedures (i.e.
operating, maintenance, testing, training), EMI/RFI, and functional diversity.
Dr. Lewis asked if there was a potential for common-mode failure due to an inaccurate internal voltage standard used for self-calibration. Mr. Mason stated that each protection channel has its own internal standard, so a failure could only affect one channel.
Mr. Mason stated that based on this audit, Zion was required to perform more accurate and more sensitive EMI/RFI mapping of their control room areas at the request of the NRC contractor performing the review of their application, and with the concurrence of the NRC staff.
Dr. Lewis questioned the contractor's rationale for this, in that a 20% uncertainty in measurement readings which were "several orders of magnitude" below environmental limits for the digital equipment did not seem to support the need for more precise measurements.
Mr. Mason agreed, but noted that they produced the-requested data to resolve the issue.
In addition, he also noted that an auditor identified some misspelled words in the comment section of non-executable code, and required them to "re-justify" the entire V&V process.
Finally, the staff required an extensive defense-in-depth analysis which evaluated the effectiveness of accident mitigation actions assuming that the entire Eagle 21 protection system fails in a non-detectable unknown state.
Mr. Mason discussed their effort to replace pneumatic and electro-mechanical EDG control relays with digital, and their current decision not to use digital technology for this upgrade primarily due to the anticipated difficulty in answering the same type of NRC staff questions, which were asked during the Eagle 21 licensing review, for the commercial products being considered for the EDG control system upgrade.
Dr. Lewis noted that the Defense Science Board has evaluated the cost benefit of using commercial electronic equipment which has seen extensive use, for defense applications, and suggested that it might have some useful insights for this present discussion.
Mr. Davis asked if they had discussed their concerns about the licensing difficulties with the NRC staff.
Mr.
Mason stated that they had not, their rationale was entirely based on perception.
Mr. Mason concluded by noting the lack of formal licensing guidance for digital upgrades, the existing uncertainty in the licensing process for these upgrades, and that the economics of this uncertainty may discourage the use of digital equipment best suited i
for the application.
l i
\\
Minutes of ACRS Subcommittee on Computers in NPP Operations i
July 21, 1993 ACRS CONSULTANT REPORT - -
Mr.
Pat Place.
Software Engineering Institute I
Mr. Place noted that the issue of defining a threshold appears to be one of continuing contention, in that the staff's proposal to.
define the threshold by system will pre-judge the determination of an unreviewed safety question.
An alternative threshold for an existing digital system may be defined, in part, by the extent of changes made to existing software.
However, he noted that even very slight sof tware changes have created very large problems (i.e.
the 1991 AT&T long distance switching failure).
He further opined that the staff's intended practice of reducing the extent of their review for digital systems which were previously licensed may not be advisable because of this fact.
Mr. Place stated that common-mode failures must distinguish between.
" design-time" f ailures which result from an inadvertent bug lef t in the final sof tware code, and " execution-time" failures which result -
from individual hardware failures.
Mr. Place expressed his opinion that the software V&V standards currently endorsed in the draft NUMARC guideline document are-not adequate to ensure sof tware quality to the level needed for safety-critical systems. He noted that U.K. Ministry of Defence standards MOD-00-55 and MOD-00-56 were much better, in that the requirements were more well-defined, and clearly mandate the use of mathematics to describe the system design.
In addition,.he supported the use of failure mode analysis for achieving safe sof tware, as opposed to perfect software.
Mr. Place pointed out that commercial item dedication is clearly a problem, in that'nearly all such items have bugs.
He also stated that attempts to define diversity must include recognizing that there is diversity in implementation (an aircraf t with both prop and jet engines) as well as diversity in function (which is NOT achieved by the aircraf t example).
He gave the example of two control systems, one of which is simple and thus easier to verify its correctness, and it maintains a mechanism within gross boundaries.
The other is much more complex and therefore its correctness may not be provable, yet it maintains the same mechanism within very precise bounds.
He offered this as en l
example of both diversity of function and diversity of implementation.
He also noted that work done by Dr. Nancy Leveson has indicated that diverse sof tware may not improve safety, in that similar logic decisions / errors are made by independent software designers trying to achieve diversity. '
,j l
=
4 i
Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 Mr.
Place offered his concern that too much self-diagnostic software may be a problem, in that as the complexity of the code-increases, so does the attendant difficulty in proving its correctness.
Further, he stated that he does not know of any method for assessing the probability of failure for sof tware, or of testing software for high levels of reliability.
In addition, he stated that software " walk-throughs" may be processes where one designer shares their misconceptions with another, thus making it risky to tely on such practices to achieve correct code.
He also noted that the NRC staff's comments on the NUMARC guidelines regarding strengthening the requirements for training and qualification on computer systems was good.
Mr.
Davis asked if a virus could strike all like software simultaneously.
Mr.
Place responded that although it was conceivable,'he felt it was highly unlikely.
STATUS OF NRC RESEARCH-INTO DIGITAL SYSTEM ENVIRONMENTAL OUALIFICATION REOUIREMENTS - Mr. Milt Vacrins, NRC RES Mr. Vagins noted that based on discussion during the previous meeting with this Subcommittee on June 16, 1993, research efforts now include the effects of smoke cr. computer-based systems.
In addition, the ACRS recommendations coming out of its October 9, 1992 meeting to prioritize the environmental stressors and learn from U.S. military experience have been incorporated into the t
research effort.
He stated that the NRC has reviewed non-classified military documents on EMI/RFI and found them not particularly relevant to nuclear power plant environments.
Dr.
r Lewis recorded his disagreement with this statement.
However, Mr.
Vagins stated that they are accessing classified military information through Scientech, Inc., and will continue their review in this area.
Dr. Lewis questioned where the EMI/RFI standard of 20 volts per meter field intensity was derived.
Mr. Ewing, RES, stated that he thought it came from ANSI standard C-63.12 or C-95.1 and that it represented a maximum " harsh industrial environment."
Mr. Vagins offered to respond later on the basis for this figure.
Mr. Vagins noted that a draft regulatory guide for EMI/RFI is out for internal review within the staff.
Development of technical bases for qualification methods for advanced I&C systems is continuing on schedule. ;
a w
. Minutes of ACRS Subcommittee on Computers in NPP Operations July.21, 1993 SUBCOMMITTEd ACTION The Subcommittee decided'to draft a letter at a later date.
FOLLOW-UP ITEMS Mr. Russell offered to include a briefing on staff insights _ gained from DoD interactions which were classified.
This has been arranged for the September full Committee meeting, together with Mr. Russell's briefing on the results of foreign trips by NRC staff.
Mr. Vagins offered to provide further information on the basis for the 20 v/m field intensity standard for EMI/RFI in nuclear power plants.
f 1
F b
I 1
t l
Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 BACV, GROUND MATERIAL PROVIDED THE SUBCOMMITTEE FOR THIS MEETING 1.
ACRS letter, Subj:
" ENVIRONMENTAL QUALIFICATION FOR DIGITAL INSTRUMENTATION AND CONTROL SYSTEMS," dated November 12, 1992.
2.
Memorandum from J.
Taylor, EDO, to P.
Shewmon, ACRS, Subj:
" ENVIRONMENTAL QUALIFICATION FOR DIGITAL INSTRUMENTATION AND CONTROL SYSTEMS," dated December 10, 1992.
3.
Excerpt f rom minutes of 393 rd ACRS meeting of January 1993, Subj:
Replacement of Analog Instrumentation with Digital Instrumentation.
- 4. NRC staff handout from presentation to ACRS on January 7,
- 1993, Subj:
Proposed Generic Letter on the Use of 50.59 for Digital Systems Replacements.
5.
NRC staff handout from presentation at Regulatory Information Conference of May 4,
1993, Subj:
Operating Reactor Analog-to-Digital Instrumentation and Control System Retrofits.
- 6. Letter f rom J. Wermiel, NRR, to A. Marion, NUMARC, dated June 2, 1993, transmitting NRC staf f comments on NUMARC proposed guidelines for reviewing digital system upgrades under 10 CFR 50.59.
7.
Memorandum from R.
Uhrig, NSRRC, to E.
Beckjord, RES, et.al.,
Subj:
" Report on Attendance at:
(1) NRC-Regulatory Information Conference, May 4,
1993, and (2) IAEA Working Group on Nuclear Power Plant Instrumentation, May 5-7, 1993" dated May 11, 1993.
9.
Excerpts from D.C.
Cook presentations to NRC staff on "Foxboro H-Line instrumentation replacementa of April 29, 1991 and December 1, 1992.
- 9. Letter f rom B. Boger, NRR, to G. Fitzpatrick, American Electric Power Co.,
Subj:
" Analog-to-Digital Instrumentation Replacement Under 10 CFR 50.59 - Donald C. Cook Nuclear Plant Units 1 and 2,"
dated August 22, 1991.
- 10. 10 CFR 50.59 NOTE:
Additional details of this meeting can be obtained from a transcript of this meeting available in the NRC Public Document Room, 2120 L Street, N.W.,
Washington, D.C.
20006, (302) 634-3274, or can be purchased from Ann Riley and Associates, Ltd.,
1612 K Street, N.W.,
Suite 300, Washington, D.C.
20006, (202) 292-3950.
l';>
+ - - -.........
m.
. s Jg. #
4 g
[
o :-
>s t-I' i ATTACHMENT i
i 1
4 k
i r
i k
a d
a h
1 c
.,,.-.-i,.,--. - - -,, -..... - - -, - -,
n, n
-s Minutes of ACRS Subcommittee on Computers in NPP Operations July 21, 1993 LIST OF ATTENDEES NRC M. Vagins, RES E. Schweibinz, Region III J. Wermiel, NRR J. Craig, RES R.
Kornasiewicz, RES J. Vora, RES J. Mauck, NRR P.
Loeser, NRR B.
Pulsifer, NRR B. Boger, NRR W.
Russell, NRR C. Thomas, NRR J.
Calvert, Region I R. Matthew, NRR L.
Beltracchi, RES F.
Coffman, RES R.
Johnson, NRR J.
Mathis, ACRS B. Wetzel, NRR J.
Kramer, RES U.S. INDUSTRY / GOVERNMENT T.
Pietrangelo, NUMARC S. Brewer, American Electric Power Service Corporation W.
Sotos, American Electric Power Service Corporation R.
Carruth, American Electric Power Service Corporation D. Hershberger, SPC/KWU R.
Fink, MPR Associates J.
Betlack, MPR Associates J. Chenard, ARC R. Torok, EPRI R. Mason, Commonwealth Edison, Co.
S.
Stimac, Commonwealth Edison, Co.
C. Willbanks, NUS R. Spence, DOE S. Horton, DOE D. Modeen, NUMARC G. Rudy, NUS D. Teague, Winston and Strawn M.
Libby, NUSMG P. Ewing, ORNL V. Glygalo, UKRSCNRS N.
Kurilchik, UKRSCNRS ATTACHMENT
- 1
-