Creating smaller fashions with a brand new sort of embedding layer

Every time coping with categorical knowledge, newcomers resort to one-hot encoding. That is typically okay, however if you’re coping with 1000’s and even hundreds of thousands of classes, this strategy turns into infeasible. This has the next causes:

1. Elevated dimensionality: For every class, you get a further function. This could result in the curse of dimensionality. The info turns into extra sparse, and the mannequin might undergo from elevated computational complexity and decreased generalization efficiency.
2. Lack of semantics: One-hot encoding treats every class as an unbiased function, ignoring any potential semantic relationships between classes. We lose significant relationships current within the unique categorical variable.

These issues happen within the space of pure language processing (we’ve got a bunch of phrases) or suggestion methods (we’ve got a bunch of consumers and/or articles) and may be overcome with the assistance of embeddings. Nonetheless, in case you have many of those embeddings, the reminiscence necessities to your mannequin can skyrocket to a number of gigabytes.

On this article, I wish to present you many methods to lower this reminiscence footprint. Certainly one of these methods comes from an fascinating paper Compositional Embeddings Utilizing Complementary Partitions for Reminiscence-Environment friendly Advice Methods by Shi et al. We can even do some experiments to see how these strategies fare in a score prediction job.

Briefly, as a substitute of lengthy, sparse vectors, we wish brief, dense vectors of some size d — our embeddings. The embedding dimension d is a hyperparameter we are able to freely select ourselves.