ML23262B196
| ML23262B196 | |
| Person / Time | |
|---|---|
| Issue date: | 06/07/2023 |
| From: | Chang Y, Friedman C, Mishkin A, Polra S, Pringle S, Tanya Smith, Vasquez G NRC/RES/DRA/HFRB, Sphere of Influence |
| To: | |
| References | |
| Download: ML23262B196 (1) | |
Text
Machine Learning Demo Wednesday Prioritizing Inspections using ML Alec Mishkin, Guillermo Vasquez, Stuti Polra, Casey Friedman, Scott Pringle, Theresa Smith Wednesday, June 7, 2023
Agenda Safety Clusters and Operating Experience Insights Topic Modeling Metrics Applied to Experiments Progress 2
Safety Clusters and Operating Experience Insights 3
Strong Clustering Performance - 3 of 4 in the same Safety Cluster using Summary 1 2 Full Item Key Finding Summary Intro Phrase Plant 1 2 61 38 Plant 2 5 1 1 3 61 Plant 3 3 1 2 Plant 4 3 1 2 4
Waterford Finding (ML22241A143) Vocab + Key Phrase Topic Representation Item Introduction The inspectors identified a finding of low to moderate safety significance (preliminary White) and apparent violations of 10 CFR 50.47(b)(4) and 10 CFR 50.47(b)(9).
Specifically, the licensee failed to maintain the reliable and accurate indications on PRM-IRE-0002, Condenser Exhaust WRGM, Mid and High Range Detectors. This 2 resulted in the potential to over classify an emergency up to a General Emergency, as well as to produce inaccurate dose assessments from January 2011 to February 4, 2022.
Summary (pegasus-cnn-dailymail)
The licensee failed to maintain the reliable and accurate indications on PRM-IRE-0003, Condenser Exhaust WRGM, Mid and High Range Detectors. This resulted in the potential to over classify an emergency up to a General Emergency, as well as to 61 produce inaccurate dose assessments from January 2011 to February 4, 2022.
After probability outlier reduction Key Phrases (KeyphraseVectorizers + Guided KeyBERT) condenser exhaust wrgm. moderate safety significance. inaccurate dose assessments.
cfr. high range detectors. accurate indications. apparent violations. prm. general emergency. inspectors. licensee. preliminary white. emergency. potential. mid. february.
38 january. finding 5
Vogtle Finding (ML20209A002) Vocab + Key Phrase Topic Representation Item Introduction The inspectors identified an Apparent Violation (AV) of Technical Specification (TS) 3.3.3, for the failure to correctly calibrate the Unit 1 (U1) and Unit 2 (U2) containment high-range area radiation monitors 1RE-0005, 1RE-0006, 2RE-0005, and 2RE-0006.
Specifically, the source-to-detector geometry used for isotopic calibrations was not fixed and reproducible which resulted in radiation monitor indications in the main 5
control room that were biased high. These radiation monitors are relied upon during an accident to provide release assessment for use by plant operators in determining the need to invoke site emergency plans.
Summary (pegasus-cnn-dailymail)
The source-to-detector geometry used for isotopic calibrations was not fixed and reproducible. These radiation monitors are relied upon during an accident to provide 1 release assessment for use by plant operators.
Key Phrases (KeyphraseVectorizers + Guided KeyBERT) radiation monitor indications. range area radiation monitors. radiation monitors. isotopic calibrations. technical specification. release assessment. vogtle unit. detector geometry.
inspectors. site emergency plans. unit. failure. apparent violation. main control room.
1 plant operators. containment. av. accident. u1. ts 6
Fermi Finding (ML17055C090)
Item Introduction An NRC identified finding preliminarily determined to be of low to moderate safety significance (White), and an associated apparent violation of Title 10 of the Code of Federal Regulations (10 CFR) 50.54(q)(2) and 10 CFR 50.47(b)(9) was identified for the licensees failure to maintain the effectiveness of its emergency plan and use adequate methods, systems, and equipment for assessing and monitoring actual or potential offsite consequences of a radiological emergency. Specifically, the licensee failed to maintain the ability to accurately declare an Emergency Action Level (EAL) classification, RG-1.1, and develop and issue accurate protective action recommendations (PARs) during the implementation of the sites Emergency Plan in response to a rapidly progressing accident. The licensee inaccurately analyzed the effect of increasing background radiation on the sites Standby Gas Treatment System accident range radiation monitor (AXM) indications based on the installed configuration of the AXM. As configured, the AXM could provide inaccurate indications of radioactive releases that are used as the licensees basis for determining EAL classification and development of PARs. The licensee documented the issue in the corrective action program as CR-16-29230, and actions were completed to restore the accuracy of the indications provided by the AXM. The inspectors determined that the licensees failure to maintain the effectiveness of its emergency plan and use adequate methods, systems, and equipment for assessing and monitoring actual or potential offsite consequences of a radiological emergency was a performance deficiency; the cause was reasonably within the licensees ability to foresee and correct; and should have been prevented. The inspectors determined the issue was more than minor because it adversely affected the emergency preparedness cornerstone objective to ensure that the licensee is capable of implementing adequate measures to protect the health and safety of the public in the event of a radiological emergency. Specifically, the finding would result in the potential over classification of an emergency event and the potential issuance of unnecessary or early PARs. The inspectors applied Inspection Manual Chapter (IMC) 0609, Appendix B, Section 5.9. to screen this finding, and determined the licensee failed to maintain the risk significant planning standard (RSPS) identified in 10 CFR 50.47(b)(9) by ensuring adequate methods, systems, and equipment for assessing and monitoring actual or potential offsite consequences of a radiological emergency condition are in use. Using Table 5.9-1, the inspectors determined the sites dose assessment process was incapable of providing technically adequate estimates of radioactive material releases to the environment or projected offsite doses in some cases (specifically a rapidly progressing accident scenario). This significance example corresponds to a Degraded RSPS Function, which is a finding of low to moderate safety significance (White).
The inspectors determined no cross-cutting aspects were associated with the performance deficiency.
7
Fermi Finding (ML17055C090) Vocab + Key Phrase Topic Representation Item Introduction An NRC identified finding [] the licensee failed to maintain the ability to accurately declare an Emergency Action Level (EAL) classification, RG-1.1, and develop and issue accurate protective action recommendations (PARs) during the implementation of the sites Emergency Plan in response to a rapidly progressing accident. The licensee inaccurately analyzed the effect of increasing background radiation on the sites Standby Gas Treatment System 3 accident range radiation monitor (AXM) indications based on the installed configuration of the AXM. As configured, the AXM could provide inaccurate indications of radioactive releases that are used as the licensees basis for determining EAL classification and development []
performance deficiency.
Summary (pegasus-cnn-dailymail)
An NRC identified finding preliminarily determined to be of low to moderate safety significance (White) The licensee failed to maintain the effectiveness of its emergency plan and use adequate methods, systems, and equipment for assessing and monitoring actual or potential offsite consequences of a radiological emergency. The licensee inaccurately analyzed the 1
effect of increasing background radiation on the sites Standby Gas Treatment System accident range radiation monitor (AXM) indications.
Key Phrases (KeyphraseVectorizers + Guided KeyBERT) standby gas treatment system accident range radiation monitor. radiological emergency condition. radiological emergency. emergency action level. moderate safety significance. dose assessment process. emergency preparedness cornerstone objective. emergency plan. safety. 2 emergency event. radioactive releases. inspection manual chapter. accident scenario.
radioactive material releases. eal classification. accurate protective action recommendations.
background radiation. indications. federal regulations. corrective action program 8
Wolf Creek Finding (ML14167A188)
Item Introduction (Initial Entry)An apparent violation of 10 CFR 50.54(q)(2) was identified involving the failure to maintain adequate methods for assessing the actual or potential consequences of a radiological emergency between September 2012 and November 2013, in accordance with the requirements of 10 CFR 50.47(b)(9). During an exercise conducted on November 13, 2012, the licensee identified that the Electronic Dose Calculation Program did not accurately calculate the consequences of a radiological release through the main vent stack with the effluent monitor in accident mode. The inaccurate Electronic Dose Calculation Program calculation was corrected on February 25, 2014.The inspectors determined the failure to maintain a dose assessment process capable of providing a technically adequate estimate of offsite dose was a performance deficiency within the licensee's control. This finding is more than minor because it was associated with the emergency response organization performance and the Facilities and Equipment cornerstone attributes. This finding was evaluated using the Emergency Preparedness Significance Determination Process and was preliminarily determined to be of low to moderate safety significance (White) because it was a degraded risk significant planning standard function. The planning standard function was degraded because between September 13, 2012, and November 8, 2013, some calculations used to assess the offsite consequences of a radiological release were inaccurate. This issue has been entered into the licensee's corrective action system as Condition Report 2013-0076247.(Inspection Report 2014502 dated 4/2/14)
(First Update)A White violation of 10 CFR 50.54(q)(2) was identified involving the failure to maintain adequate methods for assessing the actual or potential consequences of a radiological emergency between September 2012 and November 2013, in accordance with the requirements of 10 CFR 50.47(b)(9). During an exercise conducted on November 13, 2012, the licensee identified that the Electronic Dose Calculation Program did not accurately calculate the consequences of a radiological release through the main vent stack with the effluent monitor in accident mode. The inaccurate Electronic Dose Calculation Program calculation was corrected on February 25, 2014. This finding was assigned a cross-cutting aspect in the area of problem and identification associated with evaluation because the licensee failed to recognize that the issue had not been properly evaluated one year after it was identified by the licensee. This issue has been entered into the licensee's corrective action system as Condition Report 2013-0076247.The inspectors determined the failure to maintain a dose assessment process capable of providing a technically adequate estimate of offsite dose was a performance deficiency within the licensee's ability to forsee and correct. This finding is more than minor because it was associated with the emergency response organization performance and the Facilities and Equipment cornerstone attributes and adversely affected the cornerstone objective. The cornerstone objective was affected because inaccurate dose assessments may prevent the implementation of adequate measures to protect the health and safety of the public. This finding was evaluated using the Emergency Preparedness Significance Determination Process and was determined to be of low to moderate safety significance (White) because it was a degraded risk significant planning standard function. The planning standard function was degraded because between September 13, 2012, and November 8, 2013, some calculations used to assess the offsite consequences of a radiological release were inaccurate.(Inspection Report 2014503, dated 7/1/14) 9
Wolf Creek Finding (ML14167A188) Vocab + Key Phrase Topic Representation Item Introduction (Initial Entry) [] During an exercise conducted on November 13, 2012, the licensee identified that the Electronic Dose Calculation Program did not accurately calculate the consequences of a radiological release through the main vent stack with the effluent monitor in accident mode.
The inaccurate Electronic Dose Calculation Program calculation was corrected on February 25, 2014.The inspectors determined the failure to maintain a dose assessment process 3
capable of providing a technically adequate estimate of offsite dose was a performance deficiency within the licensee's control. [] consequences of a radiological release were inaccurate (Inspection Report 2014503, dated 7/1/14).
Summary (pegasus-cnn-dailymail)
An apparent violation of 10 CFR 50.54(q)(2) was identified involving the failure to maintain adequate methods for assessing the actual or potential consequences of a radiological emergency. During an exercise conducted on November 13, 2012, the licensee identified that the Electronic Dose Calculation Program did not accurately calculate the consequences of a 1
radiological release. The inaccurate Electronic Dose Calculation Program calculation was corrected on February 25, 2014.
Key Phrases (KeyphraseVectorizers + Guided KeyBERT) emergency preparedness significance determination process. dose assessment process.
inaccurate dose assessments. inaccurate electronic dose calculation program calculation.
electronic dose calculation program. moderate safety significance. radiological emergency. 2 emergency response organization performance. condition report. radiological release. accident mode. equipment cornerstone attributes. degraded risk. safety. offsite dose. cfr. requirements.
failure. adequate measures. calculations 10
Key Phrases (KeyphraseVectorizers + Guided KeyBERT)
Fermi Finding (ML17055C090)
Waterford Finding (ML22241A143) standby gas treatment system accident range radiation monitor. radiological emergency condition. radiological emergency. emergency action level.
condenser exhaust wrgm. moderate safety significance. inaccurate dose moderate safety significance. dose assessment process. emergency assessments. cfr. high range detectors. accurate indications. apparent preparedness cornerstone objective. emergency plan. safety. emergency violations. prm. general emergency. inspectors. licensee. preliminary event. radioactive releases. inspection manual chapter. accident scenario.
white. emergency. potential. mid. february. january. finding radioactive material releases. eal classification. accurate protective action recommendations. background radiation. indications. federal regulations.
corrective action program Wolf Creek Finding (ML14167A188)
Vogtle Finding (ML20209A002) emergency preparedness significance determination process. dose radiation monitor indications. range area radiation monitors. radiation assessment process. inaccurate dose assessments. inaccurate electronic monitors. isotopic calibrations. technical specification. release dose calculation program calculation. electronic dose calculation program.
assessment. vogtle unit. detector geometry. inspectors. site emergency moderate safety significance. radiological emergency. emergency response plans. unit. failure. apparent violation. main control room. plant organization performance. condition report. radiological release. accident operators. containment. av. accident. u1. ts mode. equipment cornerstone attributes. degraded risk. safety. offsite dose. cfr.
requirements. failure. adequate measures. calculations 11
Takeaways: Topic Modeling and Operating Experience Insights Automatically extracted key phrases do capture some common phenomena of safety issues related to calibration, dose assessment, and emergency action level
- More tuning of the key phrase extraction algorithm and increased coverage in the vocabulary of words and phrases used to guide the algorithm will yield even better results First layer of topic discovery focused on overarching and frequent safety issues related to different reactor systems and components Second layer of discovering common phenomena across different reactor systems or components
- Difficulty: domain expertise required to draw conclusions and find commonalities when common words/phrases are not explicitly present in the text (conversion factors, calibration, EAL)
- More tuning (of embedding model, of dimensionality reduction and cluster formation)
- Guided topic modeling approaches
- Hierarchical topic modeling approaches
- Extension of custom vocabulary of phrases that describe practices or procedures 12
Varying Inputs and Topic Representations for Cluster Formation
- 3 inputs x 4 cluster sizes = 12 experiments 3. Topic Representation (5)
- 2 BERTopic representations for all 12 experiments TF-IDF on input text in each topic cluster:
- 3 custom representations for all 12 experiments
- Stopword removal at input level and at topic representation level BERTopic MMR (diversity = 0.6)
- 2. Topic Modeling Parameters (4) BERTopic MMR + POS MMR (diversity = 0.6) + POS (NOUN, PROPN, ADJ-NOUN, ADJ-PROPN)
MMR (diversity = 0.6);
MMR (diversity = 0.6) + POS (NOUN, TF-IDF, Counts: String matching on full item-PROPN, ADJ-NOUN, ADJ-PROPN) intros in each topic cluster:
- 1. Topic Modeling Input (3) Vocabulary 1411 abbreviations + full forms + failure modes n-grams range of 1-3 Item Introduction min cluster size 10, 20, 40, 60 Key Phrases (66,325 words/phrases extracted from Item Item Introduction Summary Introductions using KeyphraseVectorizer +
15 neighbors, 5 components Guided KeyBERT with vocab of 1411 (Pegasus_cnn_dailymail model) abbreviations, full forms and failure modes)
Item Introduction Key Phrases all-MiniLM-L6-v2 (KeyphraseVectorizer + Guided KeyBERT with Vocabulary + Key Phrases custom vocab of 1411 abbreviations, full forms and (67,402) failure modes) 13
Outlier Reduction Techniques
- 1) Topic Probability
- Soft-clustering from HDBSCAN to find the best matching topic for each outlier document
- 2) Topic Distribution
- Find the most frequent topic discussed in the outlier document, and assign that topic to the outlier document
> Sliding window applied to document, c-TF-IDF of each window is computed and compared to existing topics, similarities of each window to topics are summed to create a topic distribution for the whole document
- 3) C-TF-TDF
- Find the most similar c-tf-idf topic representation to the c-tf-idf representation of the outlier document and assign that topic to the outlier document
- 4) Embedding
- Find the most similar topic embedding to the outlier documents embedding and assign that topic to the outlier document Threshold: probability or distance to control how many outliers are assigned to topics vs kept as outliers
- keeping default value of 0 for now Topic representations can be left un-changed after adding the outlier documents to existing topics, but we recompute the topic representations after adding outlier documents so that metrics can be calculated before and after outlier assignment 14
Input and Representation Testing 15
Final Metric Reviews We have currently been focusing on two different metrics.
Although we have worked on making these metrics as useful as possible there is still plenty of room for improvement Coherent Metric
- The Coherence metric utilizes a reference corpus. Since starting this project, we have increased our corpus from being the six technical documents to 295 NUREG reports and 195 Research Information letters
- From these documents there are 363856 unique unigrams, bigrams, and trigrams
- Unfortunately, the corpus still only accounts for approximately ~51% of the words that constitute topics in an example topic model
- Future work can include more time spent on webscraping to further increase the corpus Diversity Metric
- The diversity metric does not require a reference.
- It does require a trained embedding model which can be used to calculate distances
- The embedding has not been trained specifically for our purposes and thus accuracy is limited
- Future work can include training a new embedding, which could also be used for more accurate topic modelling 16
Outlier Reduction of Custom Representations Last week we showed the effect of outlier reduction on the default bertopic representation: mmr-pos Below we present the metrics for outlier reduction using our three custom representations: Vocab, Key-Phrases, and Vocab+key Phrases Coherence Scores Introduction Input Vocab Key Phrases Vocab + Key Key Phrases Input Vocab Key Phrases Vocab + Key No Reduction -1.201 -0.801 -0.804 No Reduction -1.453 -0.886 -0.877 Probability Outlier Reduction -1.207 -0.685 -0.686 Probability Outlier Reduction -1.298 -0.711 -0.717 Diversity Scores Introduction Input Vocab Key Phrases Vocab + Key Key Phrases Input Vocab Key Phrases Vocab + Key No Reduction 0.809 0.809 0.809 No Reduction 0.808 0.810 0.812 Probability Outlier Reduction 0.805 0.801 0.804 Probability Outlier Reduction 0.803 0.796 0.799 Compared to the bertopic representations, the custom representations are more significantly impacted by the outlier reduction.
Fortunately, the most significant impact is in the improvement in coherence 17
Progress 18
SOW Task Status Phase I: March 6, 2023 - April 9, 2023 Status Phase II: March 20, 2023 - May 7, 2023 Status Describe the Problem Complete Platform/system selection and installation Complete Search the Literature Complete Data acquisition and preparation Complete Select Candidates Complete Feature pipeline engineering Complete Select Evaluation Factors Complete Clustering method experimentation & selection Complete Develop evaluation factor weights Complete Cluster pipeline engineering Complete Define evaluation factor ranges Complete Anomaly detection (as needed) Not needed Perform assessment Complete Model Development, Training, Evaluation Complete Report Results Complete Test harness development Complete Deliver Trade study report Complete PoC integration and demonstration Complete Trial runs and evaluation Complete Demonstrate PoC capability Complete Phase III: April 19, 2023 - June 16, 2023 Status Live data ingestion Complete Model execution Complete Cluster evaluation Complete Critical Method documentation In progress Technical Report Document In progress Deliver final report with findings Not started 19