ML20132D905

Forwards Summary of Findings Re Evaluation of Nuclear Matls Events Database,Per 961125 Request

ML20132D905
Person / Time
Issue date:	12/18/1996
From:	Camper L NRC OFFICE OF NUCLEAR MATERIAL SAFETY & SAFEGUARDS (NMSS)
To:	Cool D NRC OFFICE OF NUCLEAR MATERIAL SAFETY & SAFEGUARDS (NMSS)
Shared Package
ML20132D909	List: ML20132D905 ML20132D916
References
SSD, NUDOCS 9612200197
Download: ML20132D905 (8)

Text

.

~ _

Dscember '18, 1996' J

MEMORANDUM TO:

Donald A. Cool, Director Division of Industrial and Medical Nuclear Safety, NMSS FROM:

Larry W. Camper, Chief

/s/

Medical, Academic, and Commercial Use Safety Branch Division of Industrial and Medical Nuclear Safety, NMSS

SUBJECT:

EVALUATION OF THE NUCLEAR MATERIALS EVENTS DATABASE As requested in your memorandum dated November 25,1996, we have evaluated the Nuclear Materials Events Database (NMED). The evaluation raised a number of concems; i

including, programming errors and inconsistencies, slow system response time, and data errors. A detailed summary of the findings is attached.

Contact:

Michele Burgess 415-5868 Attachments: As stated Distribution:

SSSS r/f IMAB r/f fi NE02-SSD7 KRamsey w"/attagents Sa yo n

DOCUMENT NAME: H:\\NMED.MEMj Te receive e cop r of this document. Indcate in the boa:p"C" = Copy without attachment / enclosure "E" = Copy with attachment / enclosure "N* = No copy OFFICE IMAB lC IMAB bbM l

l l

NAME MBurgess/mb-fylLA LCamper DATE 12/ 13 /96 12/ 6 /96 OFFICIAL RECORD COPY 7 E6 $

-. _9612200197 961218 PDR RC NRR MECENTER 88PY

~

NMED EVALUATION -lMAB

1. GENERAL PROBLEMS WITH THE SYSTEM l

Major Problem with NMED:

Document ID No. search does not work; therefore, old item numbers, old regional log numbers, and several other categories of reference numbers are not retrievable through the NMED program.

In view of the fact that a major search function in the system did not work, there is a concern that the program had not been tested very well prior to release for use in the data quality check and that additional errors may be present that did not appear in this analysis. To verify that it was not a problem with a single installation, Michele Burgess called the INEL contact (Gary Roberts 208-526-2741), to discuss the problems with retrieving the listed reference documents.

This call was made after consulting Kevin Ramsey. The problem was pointed out, and INEL tried it on their program. It seemed that INEL was not aware of the problem. INEL's comments indicated that they considered it to be a minor problem.

INEL provided the following statement for item numbers..

940000 and higher - probably willlist at least some NUDOCS numbers Also for these INEL should have hard copies, this is the 93 and 94 data (the 94/95 data was input directly, the 93 data was backfit). NOTE: For these, we were able to find some reference documents to do a data check (approx. 75% of the total records in the samples checked in IMAB's evaluation) 93000 to 93999

- there are none 92999 and lower

- probably list old item numbers / reference numbers (#2 and #3 below)

There used to be 3 old databases (for pre-93 stuff). They were put i

i together into the current database. The old item numbers are from the 3 original databases, so they do not have any meaning in the current database. The n'd item numbers do not correspond to the current item numbers. They : ;bably do not have hard copies on these. The current database contains all of the info that the original database did, so there is no more to gain there. NOTE: For these, we probably will not be able to find any reference documents to compare. (approx. 25% of the total records in the samples checked in IMAB's evaluation)

The Document ID No. search under Query Menu 1 does not work. INEL indicated that this function would normally be used to obtain additionalinformation on non-NUDOCS references.

It is also possible to find the information by doing an AD HOC table search in Access to find the corresponding 6-digit item number in the current database, then go back into NMED to look that one up. That process was attempted with one "old regional log number" (911QTX) which had been unretrievable in NMED (it had come up as a blank record). The corresponding current

NMED Evaluation item number was found to be 900650. When a search was performed in NMED on item number 900650, the program returned a blank record for this also. This process was attempted with an "old item number" The original sample item number was 900026. One of the references listed was 306021, an "old item number". The corresponding current item number for 306021 was found through the AD HOC search to be 900026, which was the orig:nal sample item number. Since this looked like a never-ending chase or that there may be some problem with the program, this method was not pursued.

i As a result of these findings, the evaluation was modified as follows:

l

1. a NUDOCS-looking number check NUDOCS

2. ens and PNs use the NRC's Agency Wide program to retrieve them

3. any non-NUDOCS number (inclu; is 9xxxxx numbers that say "old item number", "old reg log number" or any other 6 digit /chaiacter number)

DO NOT TRY TO FIND ihem, just make note that the information was not retrievable This change was discussed with Kevin Ramsey on 12/4/96.

a l

)

Qther Problems identified:

i

1. System response was VERY slow. Some examples (m= minutes, s= seconds):

l

- to bring up a record: 45 s to get to preview using the Print button: 1 m/20 s to print summary: 2 m/50 s total time from enter # to bring up to next enter #: 5 m/35 s

- to bring up record, view all screens, and print summary and some screens: 20 m/30 s to bring up record, view reference documents, write them down (usually 1 or 2 references per record, faster than printing), and use Print Button on some where there were many references, note that system was running fast (time in approximate minutes for several records done in sequence): 3,3,6,3,2,2,4,2,2,3,6,3 when system was running slow, time to enter an item number and have it display the first screen: 2 m/14 s

2. The summary print should print all of the assoc ated information available, including code definitions and licensee information.

NMED Evaluation l

3. The Print button for the summary printout is prominently marked so there is a tendency to use that. There is another button that looks like a printer that is in a toolbar, that will do a Print Screen that is MUCH faster (approx.1/4 of the time). This second way may be part of the Access Preferences, and is not readily apparent. With the slowness of the program, e

i maybe all of the systems should be set up so that this toolbar appears.

4.

Some of the records come up as a blank template. INEL says that shouldn't happen and it is a problem with the program.

5. Under some of the printouts, it says " printing page 1.", even though it is really printing more than one page. It does not change to ". page 2...", etc. Since it is not apparent that more than one page has printed, you end up going back and paging down to the next page to print that one and it says " Printing page 1. " again. Both times you really got both pages.

l

6. Under one of the printout done, it said "page 1 of 2", when there were 3 pages.

7. The program should be consistent with terminology from screen to screen. Some examples...

- the Main menu says " Search Menu..." and the submenus say " Query Menu..." (INEL's response to this was that they forgot to change the one screen when they changed the other one)

- Item No., Event item No., Document No., Reference Document, Reference Number are all used and seem to be the same.

8. There are undefined codes on the screen where there appears to be room for a more self-explanatory code or phrase. The manual did not have an abbreviation / code definition list.

Some examples..

- Under Rep Req: Mode Report says only "W'. What is "W"? (if you look on pg B-6 of the draft manual, it says that the field lists whether the report was received written or telephone, so maybe can assume that the two values would be W and T (or maybe W and P (for phone)?)

- in " Event Details" printout and on screen: Investigation: "l". What is "l"? Is it "yes"?

Pg B-3 in the draft users manual says the INVEST field is a yes/no field for indicating whether an investigation took place, the field is identified as a character field not a logical field though. things like this should be able to be spelled out on the printout. It is a simple replacement. the program finds the code in the database, and when printing, substitutes the appropriate word or phrase EQP, FCP, etc. - plenty of room on the printout to spell out what this stands for.

4 NMED Evaluation II. DATA QUALITY CHECE The following is a summary of the data quality check. In the record details, issues identified as major errors or concern are identifed by an asterisk (*).

EQE 13 = sample size 61.54% rec w/ no retrievable references (8/13 - 2,3,4,5,6,10,11,13) 38.46% rec w/ at least one retrievable reference (including EN/PN) (5/13 - 1,7,8,9,12)

= sum of 15.38% rec w/ all OK (2/13 - 9,12) 23.08% rec w/ at least one errors / concern (3/13 - 1, 7, 8) error types by record rec # # refs # ref retr

1. minor errors / concerns in rec

1. major errors /concems in rec 1

4 1

0 3

7 1

1 2

0 8

6 5

0 1

9 5

5 0

0 12 2

1 0

0 s

tot 2

4 detail - 1 1*) NMED says fire lasted 6 minutes, letter says 2 minutes, 2*) letter reports 10:30 am as time of fire, NMED lists no time even though there is a field specifically for it.

3*) letter says reported in accordance with 20.405(a)(1)(iv) & (a)(2), NMED says reporting requirements were 70.50(a) & (b)(2)(l) (note that it is a SNM license) detail - 7

1) it ic listed as an EQP event - this is correct according to the definitions in the manual, but suggest that there be a separate category for events like these, since there appeared to be no problems at all with the equipment before, during or after the event.,2) reference lists an EN but I could not find a copy, is the EN in NMED, if so why isn't it listed here?, if not, then it seems that there was a breakdown in the system that gets documents to NMED for input.

detail - 8

• 1) contained info on the retraining provided that was not in the references (may have been in the A/S report (AR 93-017) that was not retrievable) detail - 9 OK detail -12 OK

NMED Evaluation 1.EE 10 = sample size 80% rec w/ no retrievable references (8/10 - 1,2,3,4,6,7,8,9) 20% rec w/ at least one retrievable reference (including EN/PN) (2/10 - 5,10)

= sum of 00% rec w/ all OK 20% rec w/ at least one errors / concern (2/10 - 5,10) error types by record rec # # refs # ref retr

1. minor errors / concerns in rec

1. major errors / concerns in rec 5

2 1

2 1

10 2

2 1

1 l

tot 2

1 detail-5 1*) rep req - NMED says 31.5 vs. reference says source /lic cond/an -01 lic number,

2) why not include the extra info since it is short?,3) RI event report has "cause of event" checkr.;d, but the Itr doesn't have any info on cause of event detail - 10

• 1) NMED says cause was defective or failed parts, but the letter says no results of evaluations or assessments currently available - appears to contradict,2) why not include the extra info since it is short?

MQ2 18 = sample size 83.33% rec w/ no retrievable references (15/4 a - 1,2,3,4,5,6,7,8,9,10,12,13,14,15,17) 16.67% rec w/ at least one retrievable reference (including EN/PN) (3/18 - 11.16,18)

= sum of 5.56% rec w/ all OK (1/18 - 11) 11.11% rec w/ at least one errors / concern (2/18 - 16,18) error types by record rec # # refs # ref retr

1. minor errors / concerns in rec

1. major errors / concerns in rec 11 ' 1 1

0 0

16 11 10 3

2 18 1

1 1

2 tot 4

4 detail - 11 no error, but future problem - 01/01/01 seems to have been entered as a dummy date to mean that the info was not available or known. this will become a problem in the data after 1999 detail - 16 see attached Note from Dennis Serig detail - 18 1*) event date in question NMED vs. PN,2*) NMED lists 2 mci (instead of 0.0002 mci) for child's dose - dropped decimal in NMED, 3) PN says still gathering info and lic to submit written report - NMED says nothing - was there anything more?

did it get entered and not tied to this record?

NMED Evaluation General Comments on data auality check:

Due to the program problems which prevented us from retrieving any Reference Documents for a large percentage of the sample records, we were able to data quality check only 9 total records out of the intended 41. Therefore, roughly < 22% of the intended sample size was checked. An analysis should be conducted to determine how this affects the confidence level obtained by the data check. If this trend occurred during the review of the remaining Event Classes, it is likely that not enough records were able to be checked to provide a meaningful and reliable interpretation of the errors that were found in those records that were checked, it is suggested that, prior to any further data quality checks, the program be quality checked to ensure that all functions are operable. Since the system is run through two servers in series, the program speed is very slow. Due to this slowness, the time involved in searching the database was considerable. It is recommended that a contractor be considered for work at the INEL site to perform the follow-up data quality check and data correction.

lll. INFORMATION NEEDS:

i in most cases, it is expected that the information in NMED would be used to identify information on a non-routine (or ad hoc) basis, primarily in response to an event or to determine if an identified problem may have generic implications.

The following suggestions are provided:

1. The document date should be included in the Reference Documents section. This data would be more useful to the user than the date of entry. With regard to the misadministrations, there are a number of notification and reporting requirements are j

triggered by misadministrations (such as whether, when, and how the patient, physician, NRC Operations Center, or Region / Agreement State were informed) and must be completed on a schedule defined in the regulations. Since a user might want to know more than just whether or not documents were submitted, such as the average time it takes to report or whether the required schedules were met, provision of the date of the documents would allow the user to obtain this information directly from NMED. Providing this data in separate fields would allow statistical data to be obtained regarding these issues. Other useful data, particularly for events involving misadministrations, would be the postmark date and receipt dates for incoming letters.

2. The manual should provide guidance on where each type of document can be obtained.

3. In the Annual Report, the information for leaking sources and equipment problems should include the manufacturer. There are some cases where a model number is used by two different manufacturers. Reporting the manufacturer would eliminate this potential source of confusion. In addition, for many events involving a leaking source or an equipment problem, the cause of the event is not primarily due to the licensee (or user), but to a generic problem related to design or suggested use conditions. The focus in these cases would be the manufacturer.

NMED Evaluation IV. STAFF HOURS SPENT ON EVALUATIOE Approximately 72 staff-hours have been spent on the evaluation. This breaks down into the following general areas: identifying the inoperable search function, discussing it with INEL to determine if it was a problem unique to my system or a generic programming problem, analyzing the problems encountered and revising evaluation criteria then advising IMOB of parameters of our continuing evaluation (approx 18); searching / printing data from NMED (approx 18); retrieving references from NUDOCS and retrieving EN/PNs (approx 9); and evaluating data and generating written comments during the evaluation and finalizing the written report (approx 27).

1 i

1 l