Using callbacks and logging during training with gensim

Metrics to look at with callbacks

These are the metrics provided in existing gensim callback classes.

Logging

If the callbacks metrics don’t provide what you need, you might be able to get at them through logging. For example, it’s interesting to know how many documents have converged each pass, because ideally you only want to stop training when a notable majority have converged. So as well as including callbacks, we’ll enable logging in the example.

An example

In this example, I’ve downloaded text from five different novels from Project Gutenberg and split them into their chapters (an idea I’ve taken from Text Mining with R). Because it’s just an example, I’m using these books because they’ll almost certainly split into clean topics (i.e. books in this example) without needing a lot of data, and I’ll know how many topics there are in advance of training the models. If you had no idea about the number of topics you might have in your dataset, it might be worth running this a few times to get a sense of whether that drastically changes the outcome of your ongoing metrics, or even training a few models first to explore the number of topics quickly.

Here are the packages to import:

import pandas as pd
from gensim import models
import matplotlib.pyplot as plt
import logging
import re
import os

### choose the callbacks classes to import
from gensim.models.callbacks import PerplexityMetric, ConvergenceMetric, CoherenceMetric

Set up logging as well.

# The filename is the file that will be created with the log.
# If the file already exists, the log will continue rather than being overwritten.
logging.basicConfig(filename=model_callbacks.log',
                    format="%(asctime)s:%(levelname)s:%(message)s",
                    level=logging.NOTSET)

To use this code, you should have documents: a list of all documents, where each document is a string. Then if you don’t already have your corpus and dictionary, you can run this code to get them:

# Creating the term dictionary, where every unique term is assigned an index
dictionary = corpora.Dictionary(documents)

# Creating corpus using dictionary prepared above
corpus = [dictionary.doc2bow(doc) for doc in documents]

Set up the callbacks you want to use.

# Set up the callbacks loggers
perplexity_logger = PerplexityMetric(corpus=corpus, logger='shell')
convergence_logger = ConvergenceMetric(logger='shell')
coherence_cv_logger = CoherenceMetric(corpus=corpus, logger='shell', coherence = 'c_v', texts = documents)

Then train models - you could just train one, or you could try different iterations and/or topic numbers.

# List of the different iterations to try
iterations = [50, 100, 150]

# The number of passes to use - could change depending on requirements
passes = 50

for iteration in iterations:
   
    # Add text to logger to indicate new model
    logging.debug(f'Start of model: {iteration} iterations')

    # Create model - note callbacks argument uses list of created callback loggers
    model = models.ldamodel.LdaModel(corpus=corpus,
             id2word=dictionary,
             num_topics=5,
             eval_every=20,
             passes=passes,
             iterations=iteration,
            random_state=100,
            callbacks=[convergence_logger, perplexity_logger, coherence_cv_logger])

    # Add text to logger to indicate end of this model
    logging.debug(f'End of model: {iteration} iterations')

    # Save models so they aren't lost
    if not os.path.exists(f"lda_{iteration}i50p/"):
        os.makedirs(f"lda_{iteration}i50p/")

    model.save(f"lda_{iteration}i50p/lda_{iteration}i50p.model")

I created this function to go through the saved logging file and pull out the number of documents converged at each pass. It could be simplified, but has some complexity due to the possibility of having multiple attempts at a model in the log, if you’ve rerun the code a few times. This will pick up on the latest time the logs captured the relevant model.

# Function to detect relevant numbers in log
def find_doc_convergence(topic_num, iteration, log):
    # Regex to bookend log for iteration - choose last occurrence
    end_slice = re.compile(f"End of model .*? {iteration} iterations")
    end_matches = [end_slice.findall(l) for l in open(log)]
    iteration_end = [i for i, x in enumerate(end_matches) if x]
    iteration_end = iteration_end[-1]
    start_slice = re.compile(f"Start of model .*? {iteration} iterations")
    start_matches = [start_slice.findall(l) for l in open(log)]
    start_options = [i for i, x in enumerate(start_matches) if x]
    start_options = [item for item in start_options if item < iteration_end]
    iteration_start = max(start_options)
    iteration_bookends = [iteration_start, iteration_end]
    # Regex to find documents converged figures
    p = re.compile(":(\d+)\/\d")
    matches = [p.findall(l) for l in open(log)]
    matches = matches[iteration_bookends[0]:iteration_bookends[1]]
    matches = [m for m in matches if len(m) > 0]
    # Unlist internal lists and turn into numbers
    matches = [m for sublist in matches for m in sublist]
    matches = [float(m) for m in matches]
    return(matches)

Then you can use the model.metrics to get the callbacks metrics for each model, and the function above to get the metrics from the log. This creates a dataframe of all the metrics across the models.

iterations = [50, 100, 150]

all_metrics = pd.DataFrame()

for iteration in tqdm(iterations):
    model = models.ldamodel.LdaModel.load(f"lda_{iteration}i50p/lda_{iteration}i50p.model")
    df = pd.DataFrame.from_dict(model.metrics)

    df['docs_converged'] = find_doc_convergence(5, iteration, "model_callbacks.log")
    df['iterations'] = iteration
    df['topics'] = 5

    df = df.reset_index().rename(columns={'index': 'pass_num'})

    all_metrics = pd.concat([all_metrics, df])

Finally, you can visualise the results using something like this:

for metric in ['Coherence', 'Perplexity', 'Convergence', 'docs_converged']:
   
    fig, axs = plt.subplots(1, 1, figsize=(20, 7))

    # Each plot to show results for all models with the same topic number
    for i, topic_number in enumerate([5]):
        filtered_topics = all_metrics[all_metrics['topics'] == topic_number]
        for label, df in filtered_topics.groupby(['iterations']):
            print(label)
            df.plot(x='pass_num', y=metric, ax=axs, label=label)

        axs.set_xlabel(f"Pass number")
        axs.legend()
        axs.set_ylim([all_metrics[metric].min() * 0.9, all_metrics[metric].max() * 1.1])
       
    if metric == 'docs_converged':
        fig.suptitle('Documents converged', fontsize=20)
    else:
        fig.suptitle(metric, fontsize=20)

This gets you graphs like these:

Making a decision

Now that it’s done, you need to make a decision on those values. You probably want a sense of which metrics are most important to you and the minimum performance you would be happy with. However, broadly speaking when the metric evens out, you are unlikely to see much further improvement without changing something (more data, different processing), so as a rule of thumb I would look for the point at which relevant metrics flatten and use those points as your default values for further experiementation.

Looking at the graphs for the example data, you can see:

• Coherence improves sharply then flatlines after about 7 passes. No notable difference by iterations.

• Convergence is noisier, but generally decreases for the first 20 or so passes. Almost at 0 every time after 37 passes, though it might not be worth adding those extra passes for fairly small/uncertain gains.

• Perplexity is nice and flat after 5 or 6 passes. Iterations make no difference.

• Documents converged are pretty flat by 10 passes. There are 97 documents in this example (which is very low for topic modelling), so notably the 100 and 150 iteration models are basicaly seeing all documents converge, whereas 50 iterations is a bit below.

I think if you were uisng this data and decided to use 20 as your passes value and 100 as your iterations value, you’d have a good compromise between performance and training time. If you were not concerned about the convergence value you could use fewer passes, or if you were really concerned then you could use more.

There isn’t a definitive answer, but going through this step helps you to know what the trade-offs are and help idenitfy a good option for your work.