ML23262B179
| ML23262B179 | |
| Person / Time | |
|---|---|
| Issue date: | 05/04/2023 |
| From: | Chang Y, Friedman C, Mishkin A, Polra S, Pringle S, Tanya Smith, Vasquez G NRC/RES/DRA/HFRB, Sphere of Influence |
| To: | |
| References | |
| Download: ML23262B179 (1) | |
Text
Machine Learning Demo Wednesday Prioritizing Inspections using ML Thursday, May 04, 2023 Alec Mishkin, Guillermo Vasquez, Stuti Polra, Casey Friedman, Scott Pringle, Theresa Smith
Agenda 2
Topic Modeling Metrics Update Interpreting Topics with a Customized Vocabulary Automated Key Phrase Extraction Document Summary Use Case Progress and Next Steps
Metrics Update 3
One of our efforts has been identifying if metrics can be helpful in narrowing down the best parameters for topic modelling
The following slides will be presenting results on two different experiments using three different topic modelling metrics
(Metric 1) Word Pair Embedding Distance Calculates the distance of words in topics based on an external word embedding We used this sparingly, as its very slow
(Metric 2) Inversed Rank Biased Overlap Check how many of the top words in the topics match. Requires no extra resources other than the topics
(Metric 3) Topic Coherence Estimates how words in topics support each other.
Requires an extra corpus (We are still using the standard technical specs, but will be testing suggested corpuses in the following week)
D(wi,wj) = How may times words wi and wj appear in documents together D(wi) = How may times word wi appeared alone Recap 4
Experiment 1: Topic Reduction 5
Topic Reduction is the ability to algorithmically merge similar topics (starting from least frequent) to produce a fixed number of topics
We take results from the Maximal Marginal Relevance unigram models reduced from 60 topics to 5 topics
We plotted the two diversity metric results and one coherence metric results
Reducing the number of topics seem to increase the overall coherence of our model while decreasing the overall diversity.
Using 30 topics seems to be the best trade-off considering the u-mass metric has a local maxima there
These metrics are experimental and no replacement for human input.
Experiment 2: Different Modelling Techniques 6
For this experiment we have looked at three different unigram topic modelling methods Topic Modelling with Maximal Marginal Relevance Topic Modelling with Maximal Marginal Relevance with Part of Speech Topic Modelling with No representation modelling
We performed this experiment while limiting the number of clusters to see if any of the trends were robust
We only ran Word pair embedding distance calculations on topics with a minimum of 40 and 60 documents due to time constraints
If we want the most coherent topics, Topic Modelling with Maximal Marginal Relevance with Part of Speech seems the most viable IRBO has no significant difference between the different modelling techniques
Conclusion 7
We have used diversity and coherence metrics to develop base lines that we can use to continue to fine tune our topic modelling parameters
We have seen that topic reduction increases overall coherence but at the cost of diversity
We have shown that Maximal Marginal Relevance with Part of Speech provides the most coherent topics (at least given our current corpus)
Next steps Include the other reference corpuses suggested last week Include bigram/trigrams into the modelling procedures Uses these metrics to assist in our outlier analysis
Interpreting Topics with a Customized Vocabulary 8
Documents listing acronyms and their full-forms used to create an initial vocabulary ML14300A223 (656), ML17004A106 (995), Reactor Concepts (R-100) Acronyms (1050) 2436 abbreviation-full form pairs after combining the three sources and dropping duplicates
Vocabulary is created by combining the abbreviations and the full-forms into a unique list of 4,541 terms Some abbreviations have multiple different full-forms that cant be disambiguated with simple string matching Inspection findings item intros often use the abbreviations multiple times after providing the full-form once If we dont replace all abbreviations with their full-forms and only use the full-forms to match on, our terms counts will be less than the true appearance of the terms, so abbreviations are included in this vocabulary This vocabulary can be replaced with or augment the vocabulary from the NRC
Possible enhancements:
Abbreviation Detector from Allen AIs SciSpaCy package can be used to detect abbreviations and their defined full forms from text, and then replace all occurrences of the abbreviations with their full-forms Works on typical pattern like cause of the Emergency Diesel Generator (EDG) failure the EDG failure occurred on SpaCys Entity Linker can be used to link found abbreviations (entities) to a knowledge base and defined aliases Creating a Customized Vocabulary 9
Our custom vocabulary can be used to interpret the topics we discover in our text after running BERTopic Comparative approach to using MMR, POS, KeyBERTInspired or Text Generation representation models from BERTopic to interpret the found topics in slightly different ways
Extract string matches in each document, for each topic (including the documents in the outlier topic)
Using SpaCys Phrase Matcher and our custom vocabulary of terms and phrases Allowing a match on lower case text to account for casing differences between vocabulary and inspection findings text
For each topic (composed of all the documents assigned to it):
Use counts to see how many times each term or phrase from our vocabulary appears in each topic Use tf-idf representation to see how much each term or phrase from our vocabulary appears in each topic (reducing the impact of common terms that occur frequently across all topics)
For each document:
Use counts to see how many times each term or phrase from our vocabulary appears in each document
Visualize counts and tf-idf representations of vocabulary words in documents and topics with wordclouds Interpreting Topics with Custom String Matching 10
Following results are using topics learned from inspection findings item introduction summaries generated by the pegasus-cnn-dailymail model BERTopic configuration:
Generated 75 topics, with 6242 documents in the outlier topic Topic Modeling Configuration 11 all-MiniLM-L6-v2 15 neighbors, 5 components min cluster size 20 n-grams range of 1-3 MMR (diversity = 0.6)
Topic 0 - Radiation (879 documents) 12 String Matches across Documents in the Topic (using counts as weights for font size)
String Matches across Documents in the Topic (using TF-IDF representation as weights for font size)
Topic Terms from BERTopic dose - overexposure - area - rates - workers - occupational radiation safety -
rem - safety cornerstone - exposure - ability assess
Topic 1 - Emergency Diesel Generator (763 documents) 13 String Matches across Documents in the Topic (using counts as weights for font size)
String Matches across Documents in the Topic (using TF-IDF representation as weights for font size)
Topic Terms from BERTopic emergency diesel generator - edg - emergency - fuel - generators -
transfer - generator fuel oil - division - air start - oil storage
Topic 4 - Auxiliary Feedwater Pump (349 documents) 14 String Matches across Documents in the Topic (using counts as weights for font size)
String Matches across Documents in the Topic (using TF-IDF representation as weights for font size)
Topic Terms from BERTopic auxiliary feedwater pump - feedwater - auxiliary - turbine driven auxiliary - tdafw -
service - motor driven - service water pump - esw pump - bearing oil
Document in Topic 4 - Auxiliary Feedwater Pump (349 documents) 15 String Matches in Document (using counts as weights for font size)
Topic Terms from BERTopic auxiliary feedwater pump - feedwater - auxiliary - turbine driven auxiliary - tdafw -
service - motor driven - service water pump - esw pump - bearing oil Item Introduction A self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
Summary (pegasus-cnn-dailymail)
The licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
Document in Topic 3 - Residual Heat Removal (358 documents) 16 String Matches in Document (using counts as weights for font size)
Topic Terms from BERTopic residual - component cooling water - exchangers - heat removal rhr - ccw -
pump - essential - thermal performance - removal service - heat removal pump Item Introduction A self-revealed Green finding and associated non-cited violation of Technical Specification 6.8.1.a was identified when the licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode which resulted in lifting the suction relief valve for the B RHR pump and discharging reactor coolant system (RCS) water to the pressurizer relief tank (PRT).
Summary (pegasus-cnn-dailymail)
The licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode.
Custom vocabulary may not be an exhaustive list of keywords/phrases
Automated key phrase extraction in addition to a customized vocabulary will prevent us from overlooking any safety issues discussed in the text Limitations of a Customized Vocabulary 17 Topic: 63; Topic Terms from BERTopic:
chiller tripped - train escw - compressor - tripped - chiller units - control building chilled - chill - successfully restarted chiller - expansion tank - room temperature rose Item Introduction The inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase to the recirculation phase assuming a single failure. Specifically, when verifying the adequacy of design via time critical operator actions, the licensee failed to assume a more limiting single failure of valve SI-856A or SI-856B.
String Matches in Document (using counts as weights for font size)
String Matches across Documents in the Topic (using counts as weights for font size)
Automated Key Phrase Extraction 18
Unsupervised key phrase extraction
- Tokenize text into words and phrases (varying length n-grams) to get candidate keywords/phrases
- Embed document and candidate keywords/phrases using sentence transformer model
- Compute similarity between embedded documents and embedded keywords/phrases
- Retrieve N keywords/phrases that are most similar to the document Drawback: user-defined n-gram range at tokenization step usually gives grammatically incorrect or incomplete phrases that dont quite capture the main idea of the text KeyBERT 19 Extract keyphrases with part-of-speech patterns from text documents
- POS pattern: zero or more adjectives, followed by one or more nouns by default, custom patterns can be specified
- POS tagging using the spaCy English pipeline with en_core_web_sm model by default, other SpaCy models can be used Provides document - key phrase matrix to describe the frequency of key phrases in a collection of documents
- Count based matrix
- TF-IDF based matrix Can be used in BERTopic at the tokenization +
vectorizer step to extract phrases following pos patterns instead of frequent n-grams to represent topics KeyphraseVectorizers
PatternRank 20
PatternRank combines KeyphraseVectorizers and KeyBERT to find key phrases from documents that follow a pre-defined part-of-speech pattern
Can be used on the representative documents (or all documents) in a topic to discover key phrases that best describe the topic KeyphraseVectorizers KeyBERT
Customized vocabulary of key words and phrases can be added to guide the ones discovered by KeyBERT KeyBERT and Guided KeyBERT 21 KeyphraseVectorizers to extract candidates that follow defined POS pattern
Key Phrase Extraction 22 Item Introduction The inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase to the recirculation phase assuming a single failure. Specifically, when verifying the adequacy of design via time critical operator actions, the licensee failed to assume a more limiting single failure of valve SI-856A or SI-856B.
String Matches in Document (using counts as weights for font size)
Summary (pegasus-cnn-dailymail)
The licensee failed to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase to the recirculation phase assuming a single failure. Specifically, when verifying the adequacy of design via time critical operator actions, the licensee failed to assume a more limiting single failure of valve SI-856A or SI-856B.
Item Introduction Key Phrases low safety significance - cfr part - single failure - injection phase -
recirculation phase - inspectors - licensee - design control - critical operator actions - criterion iii Summary Key Phrases single failure - recirculation phase - licensee - injection phase -
critical operator actions - flow - core - adequacy - transition - design PatternRank (KeyphraseVectorizer + KeyBERT)
(not using custom vocabulary as seed words)
Key Phrase Extraction 23 Item Introduction A self-revealed Green finding and associated non-cited violation of 10 CFR 50.65(a)(4) Requirements for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, was identified for the licensee's failure to assess and manage the increase in risk that may result from maintenance activities that were performed in the switchyard house. Specifically, the licensee's failure to assess and manage the increase in risk associated with the movement of floor tiles near vibration sensitive relays on August 3, 2022, resulted in a (1) partial loss of offsite power to the 1B' startup transformer (SUT) and (2) absent the KC-2 relay setpoint, the loss of generation (i.e., runback) of approximately 10-percent power.
String Matches in Document (using counts as weights for font size)
Summary (pegasus-cnn-dailymail)
A self-revealed Green finding and associated non-cited violation of 10 CFR 50.65(a)(4) was identified for the licensee's failure to assess and manage the increase in risk that may result from maintenance activities that were performed in the switchyard house. Specifically, the licensee's failure to assess and manage the increase in risk associated with the movement of floor tiles near vibration sensitive relays on August 3, 2022 resulted in a partial loss of offsite power to the 1B Item Introduction Key Phrases maintenance - sensitive relays - nuclear power plants - maintenance activities - startup transformer - risk - offsite power - percent power -
failure - cfr Summary Key Phrases sensitive relays - green finding - risk - maintenance activities -
switchyard house - vibration - cfr - offsite power - licensee - failure PatternRank (KeyphraseVectorizer + KeyBERT)
PatternRank (KeyphraseVectorizer + KeyBERT)
(not using custom vocabulary as seed words)
Key Phrase Extraction 24 Item Introduction A self-revealed Green finding and associated non-cited violation of Technical Specification 6.8.1.a was identified when the licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode which resulted in lifting the suction relief valve for the B RHR pump and discharging reactor coolant system (RCS) water to the pressurizer relief tank (PRT)..
String Matches in Document (using counts as weights for font size)
Summary (pegasus-cnn-dailymail)
The licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode.
Item Introduction Key Phrases pressurizer relief tank - prt - green finding - reactor coolant system -
regulatory guide - residual heat removal - suction relief valve - rhr -
procedure - procedures Summary Key Phrases rhr - residual heat removal - procedure - licensee - regulatory guide -
procedures - shutdown - revision - loop - mode PatternRank (KeyphraseVectorizer + KeyBERT)
PatternRank (KeyphraseVectorizer + KeyBERT)
(not using custom vocabulary as seed words)
Document Summary Use Case 25
Document Summary UI Concept 26 The analyst has 40 or 50 documents that need to be understood quickly.
Machine generated summaries could give insights into the general content
Title:
Failure to Verify the Adequacy of the Air Pressure Regulator Setpoint Value for Containment Isolation Valves 1(2)RF026 Summary: Inspectors identified a Green finding and an associated NCV of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the failure to verify the adequacy of the air pressure regulator setpoint value for air-operated containment isolation valves 1(2)RF026.
The finding screened as of very low safety significance (Green) because it did not represent an actual open pathway.
more
Title:
Inboard Main Steam Isolation Valve Closure Time Test Acceptance Criteria Did Not Account for the Design Basis Accident Containment Back Pressure and Pneumatic Supply Operating Pressure Summary: inspectors identified a finding (FIN) of very low safety significance (Green), for the licensees failure to perform an adequate operating experience evaluation for NRC Information Notice 2017-06. this was contrary to Point Beach Procedure PI-AA-102-1001.
more
Title:
Failure to Establish Test Program to Verify Residual Heat Removal Suction Valve Capability Summary: inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from injection phase to recirculation phase.
more
Title:
Inadequate Preventative Maintenance in Residual Heat Removal Service Water System Outlet Flow Control Valves Results in Lower Bonnet (Backseat) Bushing Failure Summary: a self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions.
more Multiple summaries for each document can be generated using different models. Analysts can view their favorite and see other summaries if desired.
If additional information is needed, an entity extracted version of the document could also be displayed.
Progress 27
SOW Task Status 28 Phase I: March 6, 2023 - April 9, 2023 Status Describe the Problem Complete Search the Literature Complete Select Candidates Complete Select Evaluation Factors Complete Develop evaluation factor weights Complete Define evaluation factor ranges Complete Perform assessment Complete Report Results Complete Deliver Trade study report Complete Phase II: March 20, 2023 - May 7, 2023 Status Platform/system selection and installation Complete Data acquisition and preparation Complete Feature pipeline engineering Complete Clustering method experimentation & selection Complete Cluster pipeline engineering Complete Anomaly detection (as needed)
Not needed Model Development, Training, Evaluation Complete Test harness development Complete PoC integration and demonstration Complete Trial runs and evaluation Complete Demonstrate PoC capability Complete Phase III: April 19, 2023 - June 16, 2023 Status Live data ingestion In progress Model execution In progress Cluster evaluation In progress Critical Method documentation Not started Technical Report Document Not started Deliver final report with findings Not started
Next Steps 29
Next Steps 30 Safety Cluster Formation
- Assign outlier documents to safety clusters - use metrics to guide quality Cluster Representations
- Continue implementing string matching approaches to find effective cluster descriptions Metrics
- Assess clusters formed using full Item Introductions vs various machine generated summaries Deliverables
- Consolidate experiments, inputs, and outputs to deliverable format