View source for B13559, Responds to NRC 900606 Request for Investigation of Employee Concerns Re Adequacy of Computer Software Testing Methods at Unit.Revised Quality Software Manual Procedures,Effective Oct 1989,fully Incorporate Applicable Industry Stds

{{Adams
| number = ML20127B409
| issue date = 08/03/1990
| title = Responds to NRC 900606 Request for Investigation of Employee Concerns Re Adequacy of Computer Software Testing Methods at Unit.Revised Quality Software Manual Procedures,Effective Oct 1989,fully Incorporate Applicable Industry Stds
| author name = Mroczka E
| author affiliation = NORTHEAST NUCLEAR ENERGY CO., NORTHEAST UTILITIES
| addressee name = Martin T
| addressee affiliation = NRC OFFICE OF INSPECTION &amp; ENFORCEMENT (IE REGION I)
| docket = 05000423
| license number = 
| contact person = 
| case reference number = FOIA-91-162
| document report number = B13559, NUDOCS 9008060256
| package number = ML20126A943
| document type = CORRESPONDENCE-LETTERS, INCOMING CORRESPONDENCE
| page count = 13
}}

=Text=
{{#Wiki_filter:.f'
                                                                                                            '/
a NORTHEAST UTIMTIES                                      a:n.,,, Onc. s,ia.n si,,,i. B,n,n. Conn.ci,,,
1           UrYE5n$eNuY"~                                        P O box 270 g
                    $7[1',7.*w"~ ,,                                      HARTFORD. CONNECTICUT 06141-0270
              '  J  w  .c 4a .  .e w                                     (203) 665-5000 C
a b
August 3, 1990 Docket No. 50 423 B13559 Hr. T. T. Hartin, Administrator Region I U. S. Nuclear Regulatory Commission 475 Allendale Road King of Prussia, l'enn! ylvania 19406 Dear Mr. Martint Hillstone Nuclear Pover Station, Unit No. 3 Employee Concerns                                                     .
By {{letter dated|date=June 6, 1990|text=letter dated June 6, 1990}},1 the NRC informed Northeast Nuclear Energy Company (NNECO) of an allegation provided to an NRC inspector concerning the adequacy of computer software testing methods at HiU stone Unit No. 3.
The allegation identified three areas of concern with procedures dealing with quality related computer programs at Hillstone Unit No. 3. The {{letter dated|date=June 6, 1990|text=June 6, 1990 letter}} requested that the results of our review and disposition of l          the matter be forwarded to Region I.         NNECO hereby provides the following response to that request.
Allegation 1 The Nuclear Engineering and Operations Procedures which provide guidance l          for the development of Quality Related Computer Programs at the Hillstone i          site are inadequate in that they do not fully implement recognized industry l          standards.       For example, NE0   2.24, " Quality Software Programs" does not l
fully adhere to ANS 7.4.3.2, " Application Criteria for Programmable Digital l
Computer Systems in Safety Systems of Nuclear Pover Generating Stations."
l l
: l. Villiam Kane letter to E. J. Hroczka dated June 6, 1990.

  ,      T. T. M:rtin B13559/Pcga 2 August 3, 1990 4

===Response===
Northeast    Utilities' software        quality assurance    progran has     been      \
,        significantly upgraded over the past three years. Revised Qual../ Software Hanual    (OSM) procedures, effective October          1989, fully incorporate applicable industry standards.          Millstone Unit No. 3 has committed in          '
Northeast Utilities Quality Assurance Program (NUQAP) Topical Reporg, Revision 13, to implement the guidelines of Regulatory Luide (RG) 1.152 ,
which endorses ANS 7.4.3.2. Revision 13 is currently being reviewed by the NRC and approval is expected by September 1990.
The QSM is applied to Quality Software having applications that includes o    The design process asrociated with Category I structures, systems, or components; o    Support     of Technical Specifications       related to Category       I structures, systems, or components, or design basis analysis; o    The    verification of compliance with Technical Specifications related to design basis analyt h, when used as the sole or principal means of verifications
!            o    Support of plant licensing with respect to Category I structures, systems, or components, or design basis analysis, and o    Implementation of a sefety function of a Category I system (0S-12).         .
The guidelines of RG 1.152 and ANS          7.4.3.2 are incorporated in various procedures ;ithin the Millstone Unit No. 3 procedural system and, by reference to the NUCAP Topical Report, other t,pplicable industry standards        ,
s
,        are applied as a:Tr v tate to the utilization of softvare in the operation of all four Nortlcast '!,ilities' nuclear units.
l 3ere it seems appropriate to provide a brief explanation of the procedural system. Nuclear Engineering and Operations (NEO)          2.24,  Revision    1, references RG 1.152 (which is an endorsement of ANS 7.4.3.2) as a source document forming the basis of the procedure. The NE0 Section 2 procedures are programmatic documents in that they are neither proscriptive nor instructive. As such, NEO 2.24 is a Program description which is then implemented with subtier procedures or, in this case, the OSM.
The OSM provides the r.plementing instructions for the control of software
        - and was written to         Mress Quality software which is used to design or provide a support fun        an for the design or operation of a nuclear power plant. Several of    t'. provisions of RG 1.152 are common to activities governing all Quality         :oftware, not just Category 1 software,      and are contained in various OSM procedures. Vithin the QSM, we have written 05-12
: 2. Criteria     for Programmable Digital Computer          System Software      in Safety-Related Systems of Nuclear Power Plants (November 1985).
    -Mu

w
,~
T. T. Martin B13559/Page 3 August 3, 1990 t
which is specifically employed for the control of Category I Softvare.
0S-12 vas written to implement                                                                                                          the additional provisions of RG 1.152 applicable to Category 1 software, namely for those systems which ensure                                                                                                          ',
(1) the integrity of the reactor coolant pressure boundary, (2) the capability to shut down the reactor and maintain it in a safe condition, or (3) the capability to prevent or mitigate the consequences of accidents that could result in potential offsite exposures comparable to the 10CFR                                                                                                           .
Part 100 guidelines. Ve have only one software system at Hillstone Unit                                                                                                            ,
No. 3 that falls within the scope of 05-12, and thus, the guidelines of RG 1.152:                                                                             the Inadequate Core Cooling System (ICCS).
Northeast Nuclear Energy Company (NNECO) purchased the ICCS as a complete softvare package from Energy Incorporated, an approved supplier. The formulation of this program was conducted under their approved QA program as a qualified supplier of safety-related material. One change to the ICCS control system has been made by the Instrument and Control (I&C) Department and is discussed as part of our response to Allegation 2.
Allegation 2 The                                                           verification / validation process for computer software programs at the
                                                                                       'U.Alstone site is deficient since the verification / validation process does at                                    include a functional test with specific predetermined acceptance iteria. This is contrary to accepted ir.dustry practice.
3 sponst
                                                                                            .e                                    have investigated the allegation regarding the verification / validation process not                                                                                           including a functional test with specific predetermined acceptance criteria. Specifically, after discussion with the NRC resident inspector at Millstone Unit No. 3, we examined Software Implementation Packages (SIPS) for Quality software and control softvare from 1987 to the                                                                                        '
p:. a sen t to evaluate the adequacy of the documentation related to validation and verification of changes and the independence of reviews performed by the independent reviewers. The SIPS selected vere on the Safety Parameter Display System (SPDS), radiation monitor system, and the ICCS. The results of the examination vere that the acceptance criteria for the validation and verification tests were adequate based on the nature and complexity of the change and conformed to applicsble procedures.
Only one change to the ICCS Category I sof tware was conducted, starting in 1989.                                                                           This involved a change to an EPROM and was programmed by the supplier of                                                                                       the system, Energy Incorporated. The plant design change record (PDCR 89-015) for this change is still open. The documentation of the                                                         EPROM includes                              a    certificate of csmpliance     to the original specification for the system.                                                                                      The modified EPROM vas installed and the ICCS vas tested with predetermined acceptance criteria as documented in procedure NUSCO-930 and automated work orders (AVO) H3-89-15952 and H3-89-15953.
1

          .           .    -             -    .. .-       -                       -    ~       -
T. T. Martin B13559/Page 4 August 3, 1990 t
All of the modifications to Quality softvare that were reviewed vere done in accordance with approved procedures that                fully implemented the functional     test    and   acceptance criteria        guidelines of    Millstone      t Administrative Control Procedures (ACPs).
In a change to plant process computer softvare (non-0A) related to ICCS                  ,
communications,     Plant Incident Report       (PIR) 3-89-205 and Licensee Event Report (LER) 89-29 vere written as a result of operational problems with program "R5 - Flux and Tilting Factors". The reviewer failed to follow procedures and did not perform an adequate verification of the installation of the plant process computer software change (non-0A) related to ICCS communications      noted above. The root cause was failure to follow procedures and inadequate verification of installation of the plant process computer softvare change. This has since been corrected as documented in LER 89-29.
Allegation 3 Personnel qualifications for review of softvare are not specified by procedure. Therefore, the possibility exists for unqualified people to conduct reviews of softvare packages.

===Response===
Guidelines    for the qualification of software reviev and test personnel are reflected in Millstone ACP-0A-2.13A, which specifies that personnel shall be qualified per ACP-0A-8.16, " Training Certification, and Identification of Qualified Inspection and Testing Personnel." In the SIPS examined (as discussed above in the response to the second allegation) personnel assigned to review software vere qualified in accordance with applicable l    procedures by experience, education, or both.
General Comments Our review of these allegations has been somewhat hampered by the lack of l
l I
specific details regarding the allegations.              If for some reason the information in this letter is not fully responsive, please infor-              us so that appropriate follov-up can be undertaken.
There is some difficulty in evaluating the verification and validation process for long term software modification and development projects, some of which were initiated as early as 1983 and extending to the present.
Thus,   the projects were started before the issuance of any controls on software and are nov subj ect to the current revision of the OSM vhich implements accepted industry standards. Implementation of changes to software prior to Revision 0 of the QSM did not require documentation of the softvare changes to the extent of the current manual. Therefore, for-early changes to softvare, adherence to quality assurance requirements may not be as well documented as it is under present standards.             .

               '7. T. Martin:
_B13559/Paga 5
               -August 3,.1990 1
Ve trust the Staff finds this information satisfactory in addressing these.                          -
concerns -and we appreciate -being given the; opportunity to address       these matters.-_ Ve also- hereby confirm that none of the information in this                        (
                                                            ~
letter is subject to the provisions of 10CFR2.790. Should you require any-additional information, please do not hesitate to contact us.:
Very truly. yours, NORTHEAST NUCLEAR ENERGY COMPANY f6 E. Jy /Hr66zka jf Senior Vice President                                            .
cc     Document Control Desk                                                                         "
D. H. Jaffe, NRC Project Manager, Millstone Unit No. 3-V. J. Raymond,' Senior Resident Inspector, Millstone-Unit No. 3 V. F. Kane, NRC Region I
                                                                          . . _ ,    _       ,m., , , .

I' .
267 l
l t
si.tE0 ATiON RECEIPT                                                          t i             Thu. Ilar 15.1990                                     ALLEGATION NO.: RI-A-90-J$DO 734PN                                                  Resident Of fice No.: 8.20.1 h ne:                                                 City / St:
Confidentiality:
Was it requested?                                                       Yes         M) 1.
Yes        No Was itimtially granted?
Yes        No We it finally granted by the allegation panel?
Does a confidentiality agreement need to te sent Yes        No to the alleger?
Yes        No He a confidentiality agreement teen signed?
Yes        No Memo documenting why it was granted is attached?
Employer: NNECO                                         Position /

==Title:==
Electrician Facility: MILL 5 TONE 2                    DOCKET N0.: 50-336 50MI1ARY: Five concerns associated with: (1) improperly completed Jurveillance on the main station betteries end inadequate personnel qualificot. ions; tii) equipment tegouts:Eiii)                                                        '
[ Liv end}(v)industnel saf et.y herard in meintenance shop. The ettechment to this ellegation receipt report contains a more detailed tummary of the conte ns, elonQ with my recommendations for followup.$
NUMBER OF CONCERN 5:             5 EMPLOYEE RECEIVING All.EGATION: WILLI                                          AMw J.thRAYNOND in accordance      F             information i m of Information ACTNITY:       _J     REACTOR
                                                                                                      ~~
FUNCTloN AL ARE A: (4) .X_ Operations   (f)    thsite H&s (c) _ safe 9;aros  (g)     offsite H&S (h)     0ther Time Required to Process Request:           Man-Hours 1

s (1) Imorocerlu comoleted surveillence on the mein station bettenes_qag Inedeovete oersonnel cualifications - See 3/15/90 memo entitled Bettery                      (
OA surveillance..Ihe ellegettons are: bettery surveillence not properly completed on March 7 as evidenced by date sheets showing "normel' level and es found cell conditions noted by elleger on March 7 'et or below' low level mark; recording incorrect date on March 7; meintenance supervision       . - e.         ,
accepting out-of- tolerence values; contractors performing North 7 test not
               .,     quellfied; end, electriciens signed for work they did not do.
                                                                                                        . n Preliminary review by the SRI noted the following on North 15:                   n e The electrolyte levels on all station bettery cells were eccepteble based     N
             "       'on Inspector observations during a tour of mein station bettery rooms -              .[
                    . therefore there is no bettery operability issue et the present time;                 3
                ,' b. Based on inspector knowledge of bettery stenderd IEEE 450,it is noted            S%
that electrolyte levels above the top of the cell plates will essure adequate CopeCity.                                                                       dii;@M d
: c. Surveillence procedures SP 2736A and/or 27368 meg have been violated                  .
on Herch 7 since both procedures require workers to add water to the y
Dettery if levels are not found 'between the high and low marks";
0.
erformed en ecceptable surveillence test on L                      tierch 14 and restored bettery levels to ecceptable values. Inspector l
observations on March 15 noted most cells just above the low level merk                .
l end about a half dozen (out of 120 cells) et tha low level mark. Since there      '
was no evidence of bettery leekege, there is some question whetheb Merformed the surveillance edequetely.
Recommendation: The resident stoff should review the recently comDieted bettery surveillances to determine whicn issue are suDstantiated and what NRC followup ections ere werranted.
(2) Eouloment teocuts - See 3/15/90 mcmo entitled Togging Program - Unit
: 2. The allegations are:TBCCW pumo improperly tagged on March 15 - use of blue tog posed a worker safety problem; the wrong type of tog was used on equipment (not specified) and a legging error occurred (SPE broeker) during PMs on Merch 12; e tagging procedure violation occurred dunn0 e recent operation of the instrument etr system; and tagging problems on Umt 2 are recurrent.

s Recommendation: The resident staf f should review there legging issues to determine what issues are substantiated and what additional ections are warrented. Further input is required first from the elleger since the information provided is too non-specific for ef ficient use of resident                                                                                                                                            ,
resources.
                                  - - - - -                             .     ~
ft$jfqh .
                                    'v                                                                               N.                                                                               . ;<; .        ,
fW                           ,                                                            ' {; ,                                   .
                                                                                                                                                                        ..                   l:,
p 3.  ..
                                                                                                                          - l-
                                                                                                                       .im                                               W q:. ;.' /                          -
                                                                                                                                                                                                                       .,,,I ,
C             Recommendell.on ' "               ,
Gy            ,-
                             '}.g ,
                                                                                                                                      ,        W1','./ggygg-M%j'$. .
                              . o                                                                                                    -                                                     .    .
W MG '            1 ', o;
                                                                                                                                                    , ,_, j , yjlv*     di t.                                    l    .
f                                                                                                                       ,,
p                                                                                                                 .s
              /        R m.e..co_mmendelion:         __                  l y5               .
n                          ._                                                                                                                                                                            y
: 3) Industriel safetu he: era in meintenence shoo - See memo dated 3/2/90 and associated electrical shop load study. A study of the maintenance shop by en independent contractor confirmed discrepencies and saf ety hozords previously identified byL                                                                                             pnd shows that NU does not act on his
                                                                           '~                                  ' - ~
concerns.
Recommendetton: This issue should ref erred to the licensee through the NSC program. Specific NRC follow et e later dele should be done to essure proper closure of safety concerns and to verify integrity of new NSC program to represent the issue' l

                         . _ . .. __               _.-_    ._ _   _ _ _ _   _ . _ . -      _____..___._m _ ..-. -
    .t G,
UNIT 2 MAIEIKEAUCE SHOP ELECTRICAL MODIFICATIONS                                              ]J q
l %g 1.         All extension cord and pigtail arrangments used'to power                                            '4 permanently mounted equipment (attached to-the shop floor) should be replaced with on/off pullbox switches                                                7' and conduit connected to the equipment. There are approximately' twenty q20)~such pieces of equipment in-the shop which require th: s change.  .
j$ 2.       Cable trays which run over the-electrical mezzanine and the machine shop.mezzanina should have covers installed on both top'and bottom.
: 3.       Lighting should be installed along the north side of the-electrical crib (above shop electrical boxes) as well as below the machine shop mezzanine.
jh4. Permanently'                      attach a lighting fixture on the North wall to the_ building structure.
e jg 5.         Conduit and wiring along the north wall which appears abandoned should be tested and if disabled should be removed.
: 6.       Addition of convenience receptacles in tho' machine Dhop mezzanine area.                                                                                        ,
: 7.       Interleaf UPS requires a dedicated circuit (120 VAC).
: 8.       Emergency lights added to the lunch room and downstairs-and upstairs bathrooms.                                                                      ,
: 9.        Proto-Power recommendations (See Attached)
: 10. Provide various equipment--(See Attached) with_ dedicated-                                           ,
circuits.
                    $$11. Disconnect flourescent lights (4 banks) over the mezzanine office area should be disconnected and removed.
I e
1 1:
l
       .- . . _ _ .                         _ . _ .                         .    .,;,..__n_--..-.__,_             , , - . -   ...

r i
Recommendations The following is a list of recommended changes:
: 1.        Panel No. 6 is supplied by two separate feeder cables that splice to one        <
feeder cable connected to a fuse / disconnect. The cables (500 mcm) to the fuse / disconnect must carry the load from both parts of panel No. 6.         '.
Based on 70% demand and a cable rated for 75'C, this cable is -
undersized. Also the cable rating is less than the protective device. NU should supply each section of panel No. 6 from separate fuses /
disconnects or circuit breakers with the appropriate cable size in lieu of one disconnect.
: 2.         The plasma arc welder, normally connected to panel No. 6 was not included in the load summary. The welder is infrequently used and has been moved to the storage area adjacent to the Maintenance Shop. The welder is rated at 140A and, when used, is connected to a 125A circuit breaker on panel No. 6.
If this welder is to be used a new circuit should be provided with the -
proper rating. The circuit should be from a panel other than aanel No. 6 since the panel No. 6 feeder is at the present time heavily loac ed.
: 3.         The air handling unit / heater connected to panel No. 4 is rated at 13.8 KW/63 amps and is being supplied by a 60 amp breaker. Breaker size         s.
should be increased to accommodate the connected load and the cable size increased so that its rating is greater than the new breaker rating.
: 4.         Table 10 is a list of equipment that during the walkdown was found to be alugged into convenience receptacles and should be considered for dec icated circuits.
: 5.         Note 6 on Tables 1-9 identifies cables connected to distribution panels
!                          with no existing loads. These cables should be disconnected at the         ,
l                          distribution panel.
l l_              6.         The lunch room presently has convenience outlets divided between two l                          20A circuits. Since actual loads on the convenience circuits are not l                          known and convenience breaker tripping has occurred, the convenience l-                         circuits should be supplied from four feeder breakers in lieu of the l                          present two. Administrative controls could also be considered to limit the type and size of loads used.      .
L
           $ 7.            Emergency light (Pnl. 8-40) shown on the lighting plan for the mezzanine should be moved to the hallway to provide more light in the walkway.
y 8.            Feeder to the hydraulic lift from panel 28 should be disconnected at the panel since the lift is not operable.
l-l 4
D
                * & 'e % *

                                                                                    . ..           ~
t o
Welding receptacle-#13, panel No. 22C4-2 should be dedicated only for h9.      the 29A & 15A welding machine since the breaker is rated at 40 amps.           '
The other welding machines are rated at 48 amps.                                 .
: 10. The following cable sizes / circuit breaker sizes should be revised to ensure    '
that the cables are protected by the associated overcurrent device:
a)    Panel No.1 A, Circuit 38, SC - 14 AWG to 12 AWG.                             ,.
b)     Panel No. 28, Circuit 108,12C - 12 AWG to 10 AWG or 30A circuit breaker to 20A circuit breaker.
c)    Panel No. 6, 500 MCM Feeder - Upgrade cable rating to greater than 400A, (See Comment 1) d)    Panel No. 8, Circuit 28B, 30C - If cable is not disconnected as recommended in Table 8, then either the cable size should be increased or circuit breaker size decreased. (10 AWG or 20A bkr) e)    Panel No. 22C4 2,500 MCM Feeder - Upgrade cable rating to greater than 400A.
f)    Panel No. 6, #4/0 Feeders - Feeders should have separate protective devices (See Comment 1) or cable rating should be greater than 40nA g)    Spare circdt should be disconnected from health facility feeder since its ratii,0 of 100A exceeds the feeder rating of 65A.
b)    The 500 MCM feeder cable between the splice point for the feeder cable to Panel No. 5 and the main transformer CT should be sized     5 for 600A,
: 11. The rating of the main transformer CT's should be increased to 600A from the present 400A.
Panel No. 28, Circuit 24 - Two wires are connected to the circuit breaker.
g 12. Verify      which wire feeds the receptacle in the Health Facility and disconnect the other wire if no load can be identified.
i Load margin exists at the 208/120 volt level for panel Nos. 28,1 A,7,3, and 8. Single l      phase loads could be added upio the level of the existing highest phase load on a given panel.
i-f 1
5

                                                                ~
    -e TABLE 10 EQUIPMENT WHICH REQUIRES                                    \,
DEDICATED CIRCUlTS During the course of the walkdown, the following equipment was found to be              '
alugged into convenience receptacles, or another machine s receptacle, and should ae considered for dedicated circuits.                                                      /
LOCATION           VOLTAGE EQUIPMENT Near Lunchroom              120V Soda Machine #3 Refridgerator(2)            Lunchroom               120V Lunchroom               120V Microwave (3)
GeneralWork Area              120V Tool Grinder Weld Shop                120V Tool Grinder Machine Shop         208V,30 Marvel Saw Machine Shop              120V Grinder Machine Shop              120V Belsaw Machine Shop               120V-Tool Grinder Machine Shop               120V
 ']                           Disc Sander Machine Shop                120V Tool Grinder Machine Shop                120V-           ,
Refridgerator
    ]                                                Machine Shop                120V-t Microwave Mezzanine                 120V Refridgerator Mezzanine -               120V-Microwave
                                . Copier               Mezzanine                 120V f-                                                                               120V Ice Machine              Mezzanine f                                                     Health Facility             120V Refridgerator-Health Facility             120V Microwave
                                                     - Health Facility            120V Copier 5
r I

                        -          . . - .                     -    -.                  --                  . .             _    .               .- - ~ _ -
207 RECORD OF. ALLEGATION PANEL DECISIONS SITE: /Os @^'s A                                                               PANEL ATTENDEES:
ALLEGATION NO.: MM78-A-833                                                     Chairman - tJ.kMd6                                         \
                                                                                               ~                                                                   !
OATE: 3/2i/90                                 (Mtg.h2345)                      Branch Chief '- 6. oJeda,4c e At PRIORITY:                           High      Medium h                   Section Chief (AOC) -
SAFETY SIGNIFICANCE: Yes h- UnknownOthers - ~ f. NA 7Wr4/ 5 - --- o T' CONCURRENCE TO CLOSE00T: 00 BC SC                                                            A. Ahmo,Jo                                       l CONFIDENTIALITY GRANTED: Yes                                                                 A, /E466 (See Allegation Receipt Report) @                                                                                                             l IS THEIR A 00L FINDING:                               Yes IS CHILLING EFFECT LETTER WARRANTED:                                    Yes         No HAS CHILLING EFFECT LETTER BEEN SENT:                                    Yes          No HAS LICENSEE RESPONDED TO CHILLING EFFECT LETTER:                                              Yes       No ACTION:
155us
                    @           RI. Te -e 4 bc
                    @ R.I             Tavn xe.                                 L3- O d e e.                                                                 y.
                                                                                                                                                             .N; i                                                                                    Information rdan e with Se freedom in this ofInfor record w on
[K                                                  --                                         FOIA        - (el f9
                    @TTA over te LN                                                           b m Amt u.w,J
                                                                                                          \             U
                    . NOTES:
  - -   .,   ..           ___ _   _ . _ _ . , _ _ _ _ _ _              _ _ _ .      _ _    _      _            _        _ -             _    ,-}}