Use LLMs and Managed Vocabularies for Enhanced Similarity Fashions

The accompanying code for this tutorial is right here.

Recommender methods are how we discover a lot of the content material and merchandise we devour, most likely together with this text. A recommender system is:

“a subclass of data filtering system that gives solutions for gadgets which are most pertinent to a specific consumer.” — Wikipedia

Some examples of recommender methods we work together with usually are on Netflix, Spotify, Amazon, and social media. All of those recommender methods try to reply the identical query: given a consumer’s previous habits, what different merchandise or content material are they most probably to love? These methods generate some huge cash — a 2013 examine from McKinsey discovered that, “35 p.c of what shoppers buy on Amazon and 75 p.c of what they watch on Netflix come from product suggestions.” Netflix famously began an open competitors in 2006 providing a a million greenback prize to anybody who may considerably enhance their suggestion system. For extra info on recommender methods see this article.

Typically, there are three sorts of recommender methods: content material based mostly, collaborative, and a hybrid of content material based mostly and collaborative. Collaborative recommender methods give attention to customers’ habits and preferences to foretell what they are going to like based mostly on what different related customers like. Content material based mostly filtering methods give attention to similarity between the merchandise themselves somewhat than the customers. For more information on these methods see this Nvidia piece.

Calculating similarity between merchandise which are well-defined in a structured dataset is comparatively simple. We may establish which properties of the merchandise we expect are most essential, and measure the ‘distance’ between any two merchandise given the distinction between these properties. However what if we wish to evaluate gadgets when the one information we now have is unstructured textual content? For instance, given a dataset of film and TV present descriptions, how can we calculate that are most related?

On this tutorial, I’ll:

1. Present a fundamental similarity mannequin (no managed vocabulary) of unstructured textual content utilizing pure language processing (NLP) methods
2. Create a style record utilizing an LLM
3. Use the style record to tag movies with genres
4. Use the style tags to construct a similarity mannequin
5. Use the style tags to create a community visualization

The aim, for me, in scripting this, was to study two issues: whether or not a taxonomy (managed vocabulary) considerably improved the outcomes of a similarity mannequin of unstructured information, and whether or not an LLM can considerably enhance the standard and/or time required to assemble that managed vocabulary.

When you don’t really feel like studying the entire thing, listed below are my foremost findings:

• The fundamental NLP mannequin (with out a managed vocabulary) definitely has some issues — it typically makes use of phrases for figuring out related motion pictures that aren’t related (just like the protagonists’ first title or the placement).
• Utilizing a managed vocabulary does considerably enhance the outcomes of the similarity mannequin, a minimum of based mostly on a number of the examples I’ve been utilizing to check the fashions.
• Constructing a easy, fundamental style record utilizing an LLM is straightforward — constructing a helpful and/or detailed style taxonomy is tough i.e. it will take extra iterations or extra descriptive prompts. I ended up constructing a fast and soiled record of about 200 genres with out definitions, which labored adequate for doing easy similarity calculations.
• Even this very fundamental style record constructed utilizing an LLM has points, nonetheless. There are duplicate genres with minor spelling variations, for instance.
• Utilizing an LLM to tag the flicks and TV reveals took a really very long time. This may simply be an issue in the best way I’ve structured my code although.
• Maybe not surprisingly, the depth and breadth of the taxonomy issues. Like I stated above, constructing an in depth and descriptive taxonomy of film genres is troublesome and would require much more work than I’m keen to do for this tutorial. However relying on the use case, that degree of element won’t be crucial. I began by constructing a taxonomy of 1000’s of genres with synonyms and definitions however that had drawbacks — the tagging grew to become tougher and the similarity calculations had been typically not nearly as good. As a result of I used to be solely a pair thousand motion pictures, having a style record of 1000’s of genres simply made each film distinctive and much like nearly nothing.
• Visualizing motion pictures and genres as graphs is superior, as all the time.

We may use pure language processing (NLP) to extract key phrases from the textual content, establish how essential these phrases are, after which discover matching phrases in different descriptions. Right here is a tutorial on how to try this in Python. I gained’t recreate that complete tutorial right here however here’s a temporary synopsis:

First, we extract key phrases from a plot description. For instance, right here is the outline for the film, ‘Indiana Jones and the Raiders of the Misplaced Ark.’

“When Indiana Jones is employed by the federal government to find the legendary Ark of the Covenant, he finds himself up in opposition to the whole Nazi regime.”

We then use out-of-the-box libraries from sklearn to extract key phrases and rank their ‘significance’. To calculate significance, we use term-frequency-inverse doc frequency (tf-idf). The concept is to steadiness the frequency of the time period within the particular person movie’s description with how frequent the phrase is throughout all movie descriptions in our dataset. The phrase ‘finds,’ for instance, seems on this description, however it’s a frequent phrase and seems in lots of different film descriptions, so it’s much less essential than ‘covenant’.

This mannequin truly works very nicely for movies which have a uniquely identifiable protagonist. If we run the similarity mannequin on this movie, probably the most related motion pictures are: ‘Indiana Jones and the Temple of Doom’, ‘Indiana Jones and the Final Campaign’, and ‘Indiana Jones and the Kingdom of the Crystal Cranium’. It is because the descriptions for every of those motion pictures incorporates the phrases, ‘Indiana’ and ‘Jones’.

However there are issues right here. How do we all know the phrases which are extracted and used within the similarity mannequin are related? For instance, if I run this mannequin to search out motion pictures or TV reveals much like ‘Beavis and Butt-head Do America,” the highest result’s “Military of the Lifeless.” When you’re not a classy movie and TV buff like me, you will not be conversant in the animated sequence ‘Beavis and Butt-Head,’ that includes ‘unintelligent teenage boys [who] spend time watching tv, ingesting unhealthy drinks, consuming, and embarking on mundane, sordid adventures, which frequently contain vandalism, abuse, violence, or animal cruelty.’ The outline of their film, ‘Beavis and Butt-head Do America,’ reads, ‘After realizing that their boob tube is gone, Beavis and Butt-head set off on an expedition that takes them from Las Vegas to the nation’s capital.’ ‘Military of the Lifeless,’ however, is a Zack Snyder-directed ‘post-apocalyptic zombie heist movie’. Why is Military of the Lifeless thought of related then? As a result of it takes place in Las Vegas — each film descriptions include the phrases ‘Las Vegas’.

One other instance of the place this mannequin fails is that if I wish to discover motion pictures or TV reveals much like ‘Eat Pray Love,’ the highest result’s, ‘Extraordinarily Depraved, Shockingly Evil and Vile.’ ‘Eat Pray Love’ is a romantic comedy starring Julia Roberts as Liz Gilbert, a lately divorced girl touring the world in a journey of self-discovery. ‘Extraordinarily Depraved, Shockingly Evil and Vile,’ is a real crime drama about serial killer Ted Bundy. What do these movies have in frequent? Ted Bundy’s love curiosity can be named Liz.

These are, in fact, cherry-picked examples of instances the place this mannequin doesn’t work. There are many instances the place extracting key phrases from textual content generally is a helpful means of discovering related merchandise. As proven above, textual content that incorporates uniquely identifiable names like Energy Rangers, Indiana Jones, or James Bond can be utilized to search out different titles with those self same names of their descriptions. Likewise, if the outline incorporates details about the style of the title, like ‘thriller’ or ‘thriller’, then these phrases can hyperlink the movie to different movies of the identical style. This has limitations too, nonetheless. Some movies could use the phrase ‘dramatic’ of their description, however utilizing this technique, we’d not match these movies with movie descriptions containing the phrase ‘drama’ — we aren’t accounting for synonyms. What we actually need is to solely use related phrases and their synonyms.

How can we make sure that the phrases extracted are related? That is the place a taxonomy will help. What’s a taxonomy?

“A taxonomy (or taxonomic classification) is a scheme of classification, particularly a hierarchical classification, during which issues are organized into teams or varieties.” — Wikipedia

Maybe probably the most well-known instance of a taxonomy is the one utilized in biology to categorize all residing organisms — keep in mind area, kingdom, phylum class, order, household, genus, and species? All residing creatures could be categorized into this hierarchical taxonomy.

A word on terminology: ontologies are much like taxonomies however completely different. As this article explains, taxonomies classify whereas ontologies specify. “An ontology is the system of lessons and relationships that describe the construction of knowledge, the principles, if you’ll, that prescribe how a brand new class or entity is created, how attributes are outlined, and the way constraints are established.” Since we’re centered on classifying motion pictures, we’re going to construct a taxonomy. Nonetheless, for the needs of this tutorial, I simply want a really fundamental record of genres, which may’t even actually be described as a taxonomy. A listing of genres is only a tag set, or a managed vocabulary.

For this tutorial, we’ll focus solely on style. What we’d like is an inventory of genres that we will use to ‘tag’ every film. Think about that as a substitute of getting the film, ‘Eat Pray Love’ tagged with the phrases ‘Liz’ and ‘true’, it had been tagged with ‘romantic comedy’, ‘drama’, and ‘journey/journey’. We may then use these genres to search out different motion pictures much like Eat Pray Love, even when the protagonist just isn’t named Liz. Beneath is a diagram of what we’re doing. We use a subset of the unstructured film information, together with GPT 3.5, to create an inventory of genres. Then we use the style record and GPT 3.5 to tag the unstructured film information. As soon as our information is tagged, we will run a similarity mannequin utilizing the tags as inputs.

I couldn’t discover any free film style taxonomies on-line, so I constructed my very own utilizing a big language mannequin (LLM). I began with this tutorial, which used an LLM agent to construct a taxonomy of job titles. That LLM agent appears for job titles from job descriptions, creates definitions and tasks for every of those job titles, and synonyms. I used that tutorial to create a film style taxonomy, however it was overkill — we don’t really want to do all of that for the needs of this tutorial. We simply want a really fundamental record of genres that we will use to tag motion pictures. Right here is the code I used to create that style record.

I used Netflix film and TV present description information out there right here (License CC0: Public Area).

Import required packages and cargo english language NLP mannequin.

import openai
import os
import re
import pandas as pd
import spacy
from ipywidgets import FloatProgress
from tqdm import tqdm
# Load English tokenizer, tagger, parser and NER
nlp = spacy.load("en_core_web_sm")

Then we have to arrange our reference to OpenAI (or no matter LLM you wish to use).

os.environ["OPENAI_API_KEY"] = "XXXXXX"  # change with yours

Learn within the Netflix film information:

motion pictures = pd.read_csv("netflix_titles.csv")
motion pictures = motion pictures.pattern(n=1000) #I simply used 1000 rows of knowledge to cut back the runtime

Create a operate to foretell the style of a title given its description:

def predict_genres(movie_description):
immediate = f"Predict the highest three genres (solely genres, not descriptions) for a film with the next description: {movie_description}"
response = openai.completions.create(
mannequin="gpt-3.5-turbo-instruct",  # You should use the GPT-3 mannequin for this job
immediate=immediate,
max_tokens=50,
n=1,
cease=None,
temperature=0.2
)
predicted_genres = response.decisions[0].textual content.strip()
return predicted_genres

Now we iterate by means of our DataFrame of film descriptions, use the operate above to foretell the genres related to the film, then add them to our record of established distinctive genres.

# Create an empty record to retailer the expected genres
all_predicted_genres = []
# Create an empty set to retailer distinctive genres
unique_genres_set = set()
# Iterate by means of the film descriptions
for index, row in tqdm(motion pictures.iterrows(), whole=motion pictures.form[0]):
# Get the film description
movie_description = row['description']
# Predict the genres for the film description
predicted_genres = predict_genres(movie_description)
# Extract genres from the textual content
predicted_genres_tokens = nlp(predicted_genres)
predicted_genres_tokens = predicted_genres_tokens.textual content
# Use common expression to extract genres
genres_with_numbers = re.findall(r'd+.s*([^n]+)', predicted_genres_tokens)
# Take away main/trailing whitespaces from every style
predicted_genres = [genre.strip().lower() for genre in genres_with_numbers]
# Replace the set of distinctive genres
unique_genres_set.replace(predicted_genres)
# Convert the set of distinctive genres again to an inventory
all_unique_genres = record(unique_genres_set)

Now flip this record right into a DataFrame and save to a csv file:

all_unique_genres = pd.DataFrame(all_unique_genres,columns=['genre'])
all_unique_genres.to_csv("genres_taxonomy_quick.csv")

Like I stated, it is a fast and soiled option to generate this record of genres.

Now that we now have an inventory of genres, we have to tag every of the flicks and TV reveals in our dataset (over 8,000) with them. To have the ability to use these tags to calculate similarity between two entities, we have to tag every film and TV present with a couple of style. If we solely used one style, then all motion motion pictures might be equally related, regardless that some could also be extra about sports activities and others, horror.

First, we learn in our style record and film dataset:

#Learn in our style record
genres = pd.read_csv('genres_taxonomy_quick.csv')  # Substitute 'genres_taxonomy_quick.csv' with the precise file title
genres = genres['genre']
#Learn in our film information
motion pictures = pd.read_csv("netflix_titles.csv")
motion pictures = motion pictures.pattern(n=1000) #This takes some time to run so I did not do it for the whole dataset directly

We have already got a operate for predicting genres. Now we have to outline two extra capabilities: one for filtering the predictions to make sure that the predictions are in our established style record, and one for including these filtered predictions to the film DataFrame.

#Perform to filter predicted genres
def filter_predicted_genres(predicted_genres, predefined_genres):
# Use phrase embeddings to calculate semantic similarity between predicted and predefined genres
predicted_genres_tokens = nlp(predicted_genres)
predicted_genres_tokens = predicted_genres_tokens.textual content
# Use common expression to extract genres
genres_with_numbers = re.findall(r'd+.s*([^n]+)', predicted_genres_tokens)
# Take away main/trailing whitespaces from every style
predicted_genres = [genre.strip().lower() for genre in genres_with_numbers]
filtered_genres = []
similarity_scores = []
for predicted_genre in predicted_genres:
max_similarity = 0
best_match = None
for predefined_genre in predefined_genres:
similarity_score = nlp(predicted_genre).similarity(nlp(predefined_genre))
if similarity_score > max_similarity:  # Modify the edge as wanted
max_similarity = similarity_score
best_match = predefined_genre
filtered_genres.append(best_match)
similarity_scores.append(max_similarity)
# Type the filtered genres based mostly on the similarity scores
filtered_genres = [x for _, x in sorted(zip(similarity_scores, filtered_genres), reverse=True)]
return filtered_genres
#Perform so as to add filtered predictions to DataFrame
def add_predicted_genres_to_df(df, predefined_genres):   
# Iterate by means of the dataframe
for index, row in tqdm(df.iterrows(), whole=df.form[0]):
# Apply the predict_genres operate to the film description
predicted_genres = predict_genres(row['description'])
# Prioritize the expected genres
filtered_genres = filter_predicted_genres(predicted_genres, predefined_genres)
# Add the prioritized genres to the dataframe
df.at[index, 'predicted_genres'] = filtered_genres

As soon as we now have these capabilities outlined, we will run them on our motion pictures dataset:

add_predicted_genres_to_df(motion pictures, genres)

Now we do some information cleansing:

# Cut up the lists into separate columns with particular names
motion pictures[['genre1', 'genre2', 'genre3']] = motion pictures['predicted_genres'].apply(lambda x: pd.Collection((x + [None, None, None])[:3]))
#Hold solely the columns we'd like for similarity
motion pictures = motion pictures[['title','genre1','genre2','genre3']]
#Drop duplicates
motion pictures = motion pictures.drop_duplicates()
#Set the 'title' column as our index
motion pictures = motion pictures.set_index('title')

If we print the pinnacle of the DataFrame it ought to seem like this:

Now we flip the style columns into dummy variables — every style turns into its personal column and if the film or TV present is tagged with that style then the column will get a 1, in any other case the worth is 0.

# Mix style columns right into a single column
motion pictures['all_genres'] = motion pictures[['genre1', 'genre2', 'genre3']].astype(str).agg(','.be a part of, axis=1)
# Cut up the genres and create dummy variables for every style
genres = motion pictures['all_genres'].str.get_dummies(sep=',')
# Concatenate the dummy variables with the unique DataFrame
motion pictures = pd.concat([movies, genres], axis=1)
# Drop pointless columns
motion pictures.drop(['all_genres', 'genre1', 'genre2', 'genre3'], axis=1, inplace=True)

If we print the pinnacle of this DataFrame, that is what it appears like:

We have to use these dummy variables to construct a matrix and run a similarity mannequin throughout all pairs of films:

# If there are duplicate columns as a result of one-hot encoding, you possibly can sum them up
movie_genre_matrix = motion pictures.groupby(degree=0, axis=1).sum()
# Calculate cosine similarity 
similarity_matrix = cosine_similarity(movie_genre_matrix, movie_genre_matrix)

Now we will outline a operate that calculates probably the most related motion pictures to a given title:

def find_similar_movies(movie_name, movie_genre_matrix, num_similar_movies=3):
# Calculate cosine similarity
similarity_matrix = cosine_similarity(movie_genre_matrix, movie_genre_matrix)
# Discover the index of the given film
movie_index = movie_genre_matrix.index.get_loc(movie_name)
# Type and get indices of most related motion pictures (excluding the film itself)
most_similar_indices = np.argsort(similarity_matrix[movie_index])[:-num_similar_movies-1:-1]
# Return probably the most related motion pictures
return movie_genre_matrix.index[most_similar_indices].tolist()

Let’s see if this mannequin finds motion pictures extra much like ‘Eat Pray Love,’ than the earlier mannequin:

# Instance utilization
similar_movies = find_similar_movies("Eat Pray Love", movie_genre_matrix, num_similar_movies=4)
print(similar_movies)

The output from this question, for me, had been, ‘The Massive Day’, ‘Love Dot Com: The Social Experiment’, and ’50 First Dates’. All of those motion pictures are tagged as romantic comedies and dramas, similar to Eat Pray Love.

‘Extraordinarily Depraved, Shockingly Evil and Vile,’ the film a few girl in love with Ted Bundy, is tagged with the genres romance, drama, and crime. Essentially the most related motion pictures are, ‘The Fury of a Affected person Man’, ‘A lot Cherished’, and ‘Loving You’, all of that are additionally tagged with romance, drama, and crime. ‘Beavis and Butt-head Do America’ is tagged with the genres comedy, journey and highway journey. Essentially the most related motion pictures are ‘Pee-wee’s Massive Vacation’, ‘A Shaun the Sheep Film: Farmageddon’, and ‘The Secret Lifetime of Pets 2.’ All of those motion pictures are additionally tagged with the genres journey and comedy — there are not any different motion pictures on this dataset (a minimum of the portion I tagged) that match all three genres from Beavis and Butt-head.

You’ll be able to’t hyperlink information collectively with out constructing a cool community visualization. There are just a few methods to show this information right into a graph — we may take a look at how motion pictures are conneted through genres, how genres are linked through motion pictures, or a mixture of the 2. As a result of there are such a lot of motion pictures on this dataset, I simply made a graph utilizing genres as nodes and flicks as edges.

Right here is my code to show the information into nodes and edges:

# Soften the dataframe to unpivot style columns
melted_df = pd.soften(motion pictures, id_vars=['title'], value_vars=['genre1', 'genre2', 'genre3'], var_name='Style', value_name='GenreValue')
genre_links = pd.crosstab(index=melted_df['title'], columns=melted_df['GenreValue'])
# Create combos of genres for every title
combinations_list = []
for title, group in melted_df.groupby('title')['GenreValue']:
genre_combinations = record(combos(group, 2))
combinations_list.prolong([(title, combo[0], combo[1]) for combo in genre_combinations])
# Create a brand new dataframe from the combos record
combinations_df = pd.DataFrame(combinations_list, columns=['title', 'Genre1', 'Genre2'])
combinations_df = combinations_df[['Genre1','Genre2']]
combinations_df = combinations_df.rename(columns={"Genre1": "supply", "Genre2": "goal"}, errors="increase")
combinations_df = combinations_df.set_index('supply')
combinations_df.to_csv("genreCombos.csv")

This produces a DataFrame that appears like this:

Every row on this DataFrame represents a film that has been tagged with these two genres. We didn’t take away duplicates so there might be, presumably, many rows that seem like row 1 above — there are lots of motion pictures which are tagged as each romance and drama.

I used Gephi to construct a visualization that appears like this:

The dimensions of the nodes right here represents the variety of motion pictures tagged with that style. The colour of the nodes is a operate of a group detection algorithm — clusters which have nearer connections amongst themselves than with nodes exterior their cluster are coloured the identical.

That is fascinating to me. Drama, comedy, and documentary are the three largest nodes that means extra motion pictures are tagged with these genres than any others. The genres additionally naturally type clusters that make intuitive sense. The genres most aligned with ‘documentary’ are coloured pink and are principally some sort of documentary sub-genre: nature/wildlife, actuality TV, journey/journey, historical past, instructional, biography, and so forth. There are a core cluster of genres in inexperienced: drama, comedy, romance, coming of age, household, and so forth. One situation right here is that we now have a number of spellings of the ‘coming of age’ style — an issue I’d repair in future variations. There’s a cluster in blue that features motion/journey, fantasy, sci-fi, and animation. Once more, we now have duplicates and overlapping genres right here which is an issue. There’s additionally a small style in brown that features thriller, thriller, and horror — grownup genres typically current in the identical movie. The shortage of connections between sure genres can be fascinating — there are not any movies tagged with each ‘stand-up’ and ‘horror’, for instance.

This venture has proven me how even probably the most fundamental managed vocabulary is beneficial, and probably crucial, when constructing a content-based suggestion system. With only a record of genres we had been capable of tag motion pictures and discover different related motion pictures in a extra explainable means than utilizing simply NLP. This might clearly be improved immensely by means of a extra detailed and outline style taxonomy, but additionally by means of further taxonomies together with the forged and crew of movies, the areas, and so forth.

As is often the case when utilizing LLMs, I used to be very impressed at first at how nicely it may carry out this job, solely to be dissatisfied once I considered and tried to enhance the outcomes. Constructing taxonomies, ontologies, or any managed vocabulary requires human engagement — there must be a human within the loop to make sure the vocabulary is sensible and might be helpful in satisfying a specific use case.

LLMs and information graphs (KGs) naturally match collectively. A technique they can be utilized collectively is that LLMs will help facilitate KG creation. LLMs can’t construct a KG themselves however they will definitely allow you to create one.