ML23262B179
| ML23262B179 | |
| Person / Time | |
|---|---|
| Issue date: | 05/04/2023 |
| From: | Chang Y, Friedman C, Mishkin A, Polra S, Pringle S, Tanya Smith, Vasquez G NRC/RES/DRA/HFRB, Sphere of Influence |
| To: | |
| References | |
| Download: ML23262B179 (1) | |
Text
Machine Learning Demo Wednesday Prioritizing Inspections using ML Alec Mishkin, Guillermo Vasquez, Stuti Polra, Casey Friedman, Scott Pringle, Theresa Smith Thursday, May 04, 2023
Agenda Topic Modeling Metrics Update Interpreting Topics with a Customized Vocabulary Automated Key Phrase Extraction Document Summary Use Case Progress and Next Steps 2
Metrics Update 3
Recap One of our efforts has been identifying if metrics can be helpful in narrowing down the best parameters for topic modelling The following slides will be presenting results on two different experiments using three different topic modelling metrics (Metric 1) Word Pair Embedding Distance
- Calculates the distance of words in topics based on an external word embedding
- We used this sparingly, as its very slow (Metric 2) Inversed Rank Biased Overlap
- Check how many of the top words in the topics match. Requires no extra resources other than the topics (Metric 3) Topic Coherence
- Estimates how words in topics support each other.
- Requires an extra corpus (We are still using the standard technical specs, but will be testing suggested corpuses in the following week)
- D(wi,wj) = How may times words wi and wj appear in documents together
- D(wi) = How may times word wi appeared alone 4
Experiment 1: Topic Reduction Topic Reduction is the ability to algorithmically merge similar topics (starting from least frequent) to produce a fixed number of topics We take results from the Maximal Marginal Relevance unigram models reduced from 60 topics to 5 topics We plotted the two diversity metric results and one coherence metric results Reducing the number of topics seem to increase the overall coherence of our model while decreasing the overall diversity.
Using 30 topics seems to be the best trade-off considering the u-mass metric has a local maxima there These metrics are experimental and no replacement for human input.
5
Experiment 2: Different Modelling Techniques For this experiment we have looked at three different unigram topic modelling methods
- Topic Modelling with Maximal Marginal Relevance
- Topic Modelling with Maximal Marginal Relevance with Part of Speech
- Topic Modelling with No representation modelling We performed this experiment while limiting the number of clusters to see if any of the trends were robust We only ran Word pair embedding distance calculations on topics with a minimum of 40 and 60 documents due to time constraints If we want the most coherent topics, Topic Modelling with Maximal Marginal Relevance with Part of Speech seems the most viable IRBO has no significant difference between the different modelling techniques 6
Conclusion We have used diversity and coherence metrics to develop base lines that we can use to continue to fine tune our topic modelling parameters We have seen that topic reduction increases overall coherence but at the cost of diversity We have shown that Maximal Marginal Relevance with Part of Speech provides the most coherent topics (at least given our current corpus)
Next steps
- Include the other reference corpuses suggested last week
- Include bigram/trigrams into the modelling procedures
- Uses these metrics to assist in our outlier analysis 7
Interpreting Topics with a Customized Vocabulary 8
Creating a Customized Vocabulary Documents listing acronyms and their full-forms used to create an initial vocabulary
- ML14300A223 (656), ML17004A106 (995), Reactor Concepts (R-100) Acronyms (1050)
- 2436 abbreviation-full form pairs after combining the three sources and dropping duplicates Vocabulary is created by combining the abbreviations and the full-forms into a unique list of 4,541 terms
- Some abbreviations have multiple different full-forms that cant be disambiguated with simple string matching
- Inspection findings item intros often use the abbreviations multiple times after providing the full-form once
- If we dont replace all abbreviations with their full-forms and only use the full-forms to match on, our terms counts will be less than the true appearance of the terms, so abbreviations are included in this vocabulary
- This vocabulary can be replaced with or augment the vocabulary from the NRC Possible enhancements:
- Abbreviation Detector from Allen AIs SciSpaCy package can be used to detect abbreviations and their defined full forms from text, and then replace all occurrences of the abbreviations with their full-forms
> Works on typical pattern like cause of the Emergency Diesel Generator (EDG) failure the EDG failure occurred on
- SpaCys Entity Linker can be used to link found abbreviations (entities) to a knowledge base and defined aliases 9
Interpreting Topics with Custom String Matching Our custom vocabulary can be used to interpret the topics we discover in our text after running BERTopic
- Comparative approach to using MMR, POS, KeyBERTInspired or Text Generation representation models from BERTopic to interpret the found topics in slightly different ways Extract string matches in each document, for each topic (including the documents in the outlier topic)
- Using SpaCys Phrase Matcher and our custom vocabulary of terms and phrases
- Allowing a match on lower case text to account for casing differences between vocabulary and inspection findings text For each topic (composed of all the documents assigned to it):
- Use counts to see how many times each term or phrase from our vocabulary appears in each topic
- Use tf-idf representation to see how much each term or phrase from our vocabulary appears in each topic (reducing the impact of common terms that occur frequently across all topics)
For each document:
- Use counts to see how many times each term or phrase from our vocabulary appears in each document Visualize counts and tf-idf representations of vocabulary words in documents and topics with wordclouds 10
Topic Modeling Configuration Following results are using topics learned from Generated 75 topics, with 6242 documents in inspection findings item introduction the outlier topic summaries generated by the pegasus-cnn-dailymail model BERTopic configuration:
MMR (diversity = 0.6) n-grams range of 1-3 min cluster size 20 15 neighbors, 5 components all-MiniLM-L6-v2 11
Topic 0 - Radiation (879 documents)
Topic Terms from BERTopic dose - overexposure - area - rates - workers - occupational radiation safety -
rem - safety cornerstone - exposure - ability assess String Matches across Documents in the Topic String Matches across Documents in the Topic (using counts as weights for font size) (using TF-IDF representation as weights for font size) 12
Topic 1 - Emergency Diesel Generator (763 documents)
Topic Terms from BERTopic emergency diesel generator - edg - emergency - fuel - generators -
transfer - generator fuel oil - division - air start - oil storage String Matches across Documents in the Topic String Matches across Documents in the Topic (using counts as weights for font size) (using TF-IDF representation as weights for font size) 13
Topic 4 - Auxiliary Feedwater Pump (349 documents)
Topic Terms from BERTopic auxiliary feedwater pump - feedwater - auxiliary - turbine driven auxiliary - tdafw -
service - motor driven - service water pump - esw pump - bearing oil String Matches across Documents in the Topic String Matches across Documents in the Topic (using counts as weights for font size) (using TF-IDF representation as weights for font size) 14
Document in Topic 4 - Auxiliary Feedwater Pump (349 documents)
Topic Terms from BERTopic auxiliary feedwater pump - feedwater - auxiliary - turbine driven auxiliary - tdafw -
service - motor driven - service water pump - esw pump - bearing oil String Matches in Document Item Introduction (using counts as weights for font size)
A self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
Summary (pegasus-cnn-dailymail)
The licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
15
Document in Topic 3 - Residual Heat Removal (358 documents)
Topic Terms from BERTopic residual - component cooling water - exchangers - heat removal rhr - ccw -
pump - essential - thermal performance - removal service - heat removal pump Item Introduction String Matches in Document A self-revealed Green finding and associated non-cited violation of Technical (using counts as weights for font size)
Specification 6.8.1.a was identified when the licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode which resulted in lifting the suction relief valve for the B RHR pump and discharging reactor coolant system (RCS) water to the pressurizer relief tank (PRT).
Summary (pegasus-cnn-dailymail)
The licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the licensee failed to implement a procedure for placing the B residual heat removal (RHR) loop in shutdown cooling mode.
16
Limitations of a Customized Vocabulary String Matches across Documents in the Topic String Matches in Document (using counts as weights for font size) (using counts as weights for font size)
Item Introduction The inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase to the recirculation phase assuming a single failure. Specifically, when Topic: 63; Topic Terms from BERTopic: verifying the adequacy of design via time critical operator actions, the chiller tripped - train escw - compressor - tripped - chiller units - control building licensee failed to assume a more limiting single failure of valve SI-856A or chilled - chill - successfully restarted chiller - expansion tank - room temperature rose SI-856B.
Custom vocabulary may not be an exhaustive list of keywords/phrases Automated key phrase extraction in addition to a customized vocabulary will prevent us from overlooking any safety issues discussed in the text 17
Automated Key Phrase Extraction 18
KeyBERT KeyphraseVectorizers Unsupervised key phrase extraction Extract keyphrases with part-of-speech patterns
- Tokenize text into words and phrases (varying length n- from text documents grams) to get candidate keywords/phrases - POS pattern: zero or more adjectives, followed by one or
- Embed document and candidate keywords/phrases more nouns by default, custom patterns can be specified using sentence transformer model - POS tagging using the spaCy English pipeline with
- Compute similarity between embedded documents and en_core_web_sm model by default, other SpaCy models embedded keywords/phrases can be used
- Retrieve N keywords/phrases that are most similar to the document Provides document - key phrase matrix to Drawback: user-defined n-gram range at describe the frequency of key phrases in a tokenization step usually gives grammatically collection of documents incorrect or incomplete phrases that dont quite - Count based matrix capture the main idea of the text - TF-IDF based matrix Can be used in BERTopic at the tokenization +
vectorizer step to extract phrases following pos patterns instead of frequent n-grams to represent topics 19
PatternRank KeyphraseVectorizers KeyBERT PatternRank combines KeyphraseVectorizers and KeyBERT to find key phrases from documents that follow a pre-defined part-of-speech pattern Can be used on the representative documents (or all documents) in a topic to discover key phrases that best describe the topic 20
KeyBERT and Guided KeyBERT KeyphraseVectorizers to extract candidates that follow defined POS pattern Customized vocabulary of key words and phrases can be added to guide the ones discovered by KeyBERT 21
Key Phrase Extraction String Matches in Document PatternRank (KeyphraseVectorizer + KeyBERT)
(using counts as weights for font size)
(not using custom vocabulary as seed words)
Item Introduction The inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Item Introduction Key Phrases Control, for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase low safety significance - cfr part - single failure - injection phase -
to the recirculation phase assuming a single failure. Specifically, when recirculation phase - inspectors - licensee - design control - critical verifying the adequacy of design via time critical operator actions, the operator actions - criterion iii licensee failed to assume a more limiting single failure of valve SI-856A or SI-856B.
Summary (pegasus-cnn-dailymail)
Summary Key Phrases The licensee failed to ensure core cooling flow could be maintained and not interrupted during the transition from the injection phase to the recirculation single failure - recirculation phase - licensee - injection phase -
phase assuming a single failure. Specifically, when verifying the adequacy of critical operator actions - flow - core - adequacy - transition - design design via time critical operator actions, the licensee failed to assume a more limiting single failure of valve SI-856A or SI-856B.
22
Key Phrase Extraction String Matches in Document PatternRank (KeyphraseVectorizer + KeyBERT)
(using counts as weights for font size)
(not using custom vocabulary as seed words)
Item Introduction A self-revealed Green finding and associated non-cited violation of 10 CFR 50.65(a)(4) Requirements for Monitoring the Effectiveness of Maintenance at Nuclear Power Plants, was identified for the licensee's failure to assess and Item Introduction Key Phrases manage the increase in risk that may result from maintenance activities that were maintenance - sensitive relays - nuclear power plants - maintenance performed in the switchyard house. Specifically, the licensee's failure to assess and activities - startup transformer - risk - offsite power - percent power -
manage the increase in risk associated with the movement of floor tiles near failure - cfr vibration sensitive relays on August 3, 2022, resulted in a (1) partial loss of offsite power to the 1B' startup transformer (SUT) and (2) absent the KC-2 relay setpoint, the loss of generation (i.e., runback) of approximately 10-percent power.
Summary (pegasus-cnn-dailymail)
A self-revealed Green finding and associated non-cited violation of 10 CFR Summary Key Phrases 50.65(a)(4) was identified for the licensee's failure to assess and manage the increase in risk that may result from maintenance activities that were performed in sensitive relays - green finding - risk - maintenance activities -
the switchyard house. Specifically, the licensee's failure to assess and manage the switchyard house - vibration - cfr - offsite power - licensee - failure increase in risk associated with the movement of floor tiles near vibration sensitive relays on August 3, 2022 resulted in a partial loss of offsite power to the 1B 23
Key Phrase Extraction String Matches in Document PatternRank (KeyphraseVectorizer + KeyBERT)
(using counts as weights for font size)
(not using custom vocabulary as seed words)
Item Introduction A self-revealed Green finding and associated non-cited violation of Technical Item Introduction Key Phrases Specification 6.8.1.a was identified when the licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, pressurizer relief tank - prt - green finding - reactor coolant system -
Appendix A. Specifically, the licensee failed to implement a procedure for placing regulatory guide - residual heat removal - suction relief valve - rhr -
the B residual heat removal (RHR) loop in shutdown cooling mode which resulted procedure - procedures in lifting the suction relief valve for the B RHR pump and discharging reactor coolant system (RCS) water to the pressurizer relief tank (PRT)..
Summary (pegasus-cnn-dailymail)
Summary Key Phrases The licensee failed to implement written procedures covering activities recommended in Regulatory Guide 1.33, Revision, 2, Appendix A. Specifically, the rhr - residual heat removal - procedure - licensee - regulatory guide -
licensee failed to implement a procedure for placing the B residual heat removal procedures - shutdown - revision - loop - mode (RHR) loop in shutdown cooling mode.
24
Document Summary Use Case 25
Document Summary UI Concept The analyst has 40 or 50 documents that need to be understood quickly. If additional information is needed, an Multiple summaries for each document can be entity extracted version of the document generated using different models. Analysts can view Machine generated summaries could
Title:
Failure to Verify the Adequacy of the Air Pressure Regulator Setpoint Value for could also be displayed.
their favorite and1(2)RF026 see other summaries if desired.
give insights into the general content Containment Isolation Valves Summary: Inspectors identified a Green finding and an associated NCV of 10 CFR Part 50, Appendix B, Criterion III, Design Control, for the failure to verify the adequacy of the air pressure regulator setpoint value for air-operated containment isolation valves 1(2)RF026.
The finding screened as of very low safety significance (Green) because it did not represent an actual open pathway.
more
Title:
Inboard Main Steam Isolation Valve Closure Time Test Acceptance Criteria Did Not Account for the Design Basis Accident Containment Back Pressure and Pneumatic Supply Operating Pressure Summary: inspectors identified a finding (FIN) of very low safety significance (Green), for the licensees failure to perform an adequate operating experience evaluation for NRC Information Notice 2017-06. this was contrary to Point Beach Procedure PI-AA-102-1001.
more
Title:
Failure to Establish Test Program to Verify Residual Heat Removal Suction Valve Capability Summary: inspectors identified a finding of very low safety significance (Green) and a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control for the licensees failure to ensure core cooling flow could be maintained and not interrupted during the transition from injection phase to recirculation phase.
more
Title:
Inadequate Preventative Maintenance in Residual Heat Removal Service Water System Outlet Flow Control Valves Results in Lower Bonnet (Backseat) Bushing Failure Summary: a self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions.
more 26
Progress 27
SOW Task Status Phase I: March 6, 2023 - April 9, 2023 Status Phase II: March 20, 2023 - May 7, 2023 Status Describe the Problem Complete Platform/system selection and installation Complete Search the Literature Complete Data acquisition and preparation Complete Select Candidates Complete Feature pipeline engineering Complete Select Evaluation Factors Complete Clustering method experimentation & selection Complete Develop evaluation factor weights Complete Cluster pipeline engineering Complete Define evaluation factor ranges Complete Anomaly detection (as needed) Not needed Perform assessment Complete Model Development, Training, Evaluation Complete Report Results Complete Test harness development Complete Deliver Trade study report Complete PoC integration and demonstration Complete Trial runs and evaluation Complete Demonstrate PoC capability Complete Phase III: April 19, 2023 - June 16, 2023 Status Live data ingestion In progress Model execution In progress Cluster evaluation In progress Critical Method documentation Not started Technical Report Document Not started Deliver final report with findings Not started 28
Next Steps 29
Next Steps Safety Cluster Formation
- Assign outlier documents to safety clusters - use metrics to guide quality Cluster Representations
- Continue implementing string matching approaches to find effective cluster descriptions Metrics
- Assess clusters formed using full Item Introductions vs various machine generated summaries Deliverables
- Consolidate experiments, inputs, and outputs to deliverable format 30