Text

Meeting Slides 20230412-final
	ML23262B168
Person / Time
Issue date:	04/12/2023
From:	Chang Y, Friedman C, Mishkin A, Polra S, Pringle S, Tanya Smith, Vasquez G; NRC/RES/DRA/HFRB, Sphere of Influence
To:	;
References
	Download: ML23262B168 (1)
	v • d • e

Machine Learning Demo Wednesday Prioritizing Inspections using ML Wednesday, April 12, 2023 Alec Mishkin, Guillermo Vasquez, Stuti Polra, Casey Friedman, Scott Pringle, Theresa Smith

Agenda 2

BERTopic Jupyter Notebook Demo Named Entity Extraction Varying Text Embeddings in BERTopic Topic Modeling for Different Categories Progress and Next Steps

BERTopic Demo 3

Language models that have been trained to recognize text spans as different kinds of named entities

Most available pre-trained models can extract 4 or 18 names entities class NER models: person, location, organization, miscellaneous class NER models: cardinal value, date, event, building or facility, geo-political entity, language, law, location, money, nationality, religious or political affiliation, ordinal, organization, percent, person, product, quantity, time, work of art

These models can be trained to extract custom entities, but this would require training with a large text corpora with entities and their corresponding text spans labeled

For our use case, a pre-trained model that does a good job of extracting the entities of interest is sufficient Named Entity Recognition (NER) 4

Flairs ner-english-ontonotes-large model SpaCys en_core_web_trf model Named Entity Recognition Example 5

Flairs ner-english-ontonotes-large model SpaCys en_core_web_trf model Named Entity Recognition Example 6

Different models will be better at recognizing different entities (ex. Law with Flair, Facility with Spacy, Product with both)

Pre-trained NER models arent completely reliable on technical data, they may miss some parts of the entities or some entities altogether

Extracted entities can be used at various points in the BERTopic pipeline

- As categories to view found topics under, and extract topic representations for

- Steer dimensionality reduction of document embeddings closer to embeddings of extracted entities in a semi-supervised topic modeling approach

- With more reliable and accurate NER, and customized code:

Select which text is fed to the embedding step (along with POS tagging to select patterns of interest)

Tokenize text before c-TF-IDF step (along with POS tagging to select patterns of interest)

Named Entity Recognition (NER) 7

BERTopic Experiments Overview 8

BERTopic offers modularity at each step of the process

- Embedding

- Dimensionality Reduction

- Clustering

- Tokenizer

- Weighing scheme

- Representation tuning Each component can be easily swapped according to the goals and to accommodate the data BERTopic: Modularity 9

Topic Representation Maximal marginal relevance to reduce redundant keywords KeyBERT inspired approach to find keywords that are closely related to the representative documents of each topic Rule-based part-of-speech matching to find keywords or key phrases from representative documents that follow a specified part-of-speech pattern (nouns, adjectives followed by nouns)

Using text generation models to label topics by providing a prompt with the keywords and representative documents Chaining multiple topic representation approaches MMR KeyBERT, MMR POS MMR KeyBERT Text Generation, MMR POS Text Generation

Text Embedding Varying the language models used to embed document text Models with larger token limits, those without token limits Models that perform character level embeddings to better capture technical language

Topic Modeling per Category Perform topic modeling for each known category of the data Regions, reactor sites, reactor units, cornerstone areas, cross-cutting areas BERTopic Experiments Overview 10

Text Embedding Experimentation 11

An Embedding is the First LEGO piece in the BERTopic model

The chosen embedding converts text into a numerical format that computers can understand

When a person tries to understand a problem, they bring in their own bias and experiences

Similarly, the Embeddings give the BERTopic model the context and experience necessary to identify topics.

Just how two different people can approach the same problem differently, two different Embeddings can affect the topics identified by BERTopic

The next few slides will summarize a few of the many possible Embeddings we are using and present a few results Embeddings 12

Example Embeddings 13

BERTopics modularity allows us to very easily test different embeddings from different online repositories Embedding Description all-MiniLM-L6-v2 Designed for general purpose and speed. Trained on a large corpus of online data all-mpnet-base-v2 Designed for general purpose and quality. Trained on a large corpus of online data xlnet-base-cased Designed to work on language tasks that involve long context.

SPECTER Trained on scientific citations and designed to estimate the similarity of two publications multi-qa-MiniLM-L6-dot-v1 Model was designed to find relevant passages from specific queries.

Trained on a large and diverse set of (question, answer) pairs

Results

182 topics found (including outlier topic)

The top 6 most common results were all-MiniLM-L6-v2 14