New. Complete. Extendable.

A big share of datasets include categorical options. For instance, out of 665 datasets on the UC Irvine Machine Studying Repository [1], 42 are absolutely categorical and 366 are reported as combined. Nonetheless, distance-based ML fashions require options in a numerical format. Categorical encoders substitute the classes in such options with actual numbers.

Quite a lot of categorical encoders exist, however there have been few makes an attempt to check them on many datasets, with numerous ML fashions, and in several pipelines. This text is about one of many newest benchmarks of encoders from our latest publication [2] (poster, code on GitHub). On this story, I concentrate on the content material that enhances the publication and is of sensible significance. Specifically, past the abstract of our benchmark outcomes, I’ll:

• Present an inventory of 55 categorical encoders and the hyperlinks to seek out their explanations and implementations for many of them.
• Clarify which you can additionally use our code as a complement to the Class Encoderspython module for the encoders not but carried out there.
• Categorize the encoders into households so that you simply should not have to recollect every particular person encoder, however as a substitute have an concept of how one can construct a member of every household.
• Clarify how one can reuse the code from [2] and detailed benchmark knowledge to incorporate your encoder, dataset, or ML fashions within the comparability with out having to re-run the prevailing experiments. Relying on the scope of your experiments and your computational assets, this will prevent weeks of computation.

Why one other benchmark?

There are already a number of scientific research evaluating categorical encoders [3–12] and at the least one categorical encoder benchmark on TDS [13]. The research [2] that I’ll current right here differs primarily in scope: We in contrast consultant encoders in several configurations from a wide range of encoder households. We experimented with 5 ML fashions (choice tree, kNN, SVM, logistic regression, LGBM), 4 high quality metrics (AUC, accuracy, balanced accuracy, F1-score), 3 tuning methods (which I’ll describe shortly), 50 datasets, and 32 encoder configurations.

The next desk exhibits the encoders lined in our benchmark (the darkish inexperienced column) and in different experimental comparisons (the sunshine inexperienced columns). The blue columns present the encoders described in two further sources: on Medium [14] and within the article devoted to distinction encoders [15]. The final yellow column exhibits the encoders lined by the Class Encoders module [16]. Be aware that the code from [2] implements some encoders — from the similarity, binning, and knowledge constraining households — that aren’t a part of the Class Encoders module. As well as, we’ve discovered that the interface to the GLMM encoder carried out in R and utilized in [2] is far quicker than the GLMM encoder from Class Encoders. Subsequently, chances are you’ll discover our code helpful for these implementations.

Desk 1 already incorporates plenty of encoders, and the record is in no way exhaustive. To navigate the encoder panorama, it’s subsequently helpful to categorise encoders as a way to perceive the ideas of encoder design, fairly than to memorize a lot of particular person encoders.

Households of encoders

Within the following, we think about a categorical function of size n with cardinality ok. On the high degree, categorical encoders are supervised or unsupervised.

1. Unsupervised encoders don’t embrace the goal variable within the encoding course of.

1.1. Identifier encoders rework categorical variables utilizing a bijective perform, i.e., they assign every class a novel numeric worth or a novel mixture of numeric values. They create from 1 as much as ok new options. For instance, One-Scorching encoder creates ok options, label or ordinal encoders create a single new function, Base N encoders create ⌈log(ok)⌉ new options, the place the logarithm is of base N.

These encoders are helpful for categorical variables that denote distinctive identifiers, similar to product codes or zip codes. Sometimes, except ordinal encoder, identifier encoders don’t assume that any inherent order within the classes conveys significant info for evaluation, and thus ignore any such order through the encoding course of.

1.2. Distinction encoders rework categorical variables by assigning numerical values primarily based on comparisons between classes. Sometimes, a set of k-1 new options represents a categorical variable with ok classes. Not like identifier encoders, these encoders are particularly designed to discover the relationships between completely different ranges of a categorical variable in a regression evaluation.

To create a distinction encoder, one has a alternative of various coding schemes [15]. Every contrasts a class with different classes in a specific strategy to take a look at a associated speculation concerning the knowledge. For instance, Helmert coding contrasts every degree with the imply of subsequent ranges, whereas sum coding contrasts every degree with the general imply.

1.3. Frequency encoders substitute the classes of a categorical variable with corresponding values which are a perform of the frequency of these classes within the knowledge set. Not like identifier or distinction encoders, frequency encoders don’t assure to assign distinctive identifiers and create a single new function. They supply a numerical illustration of prevalence charges, which may be significantly helpful when an ML mannequin advantages from understanding the commonality of class values. All three frequency encoders in Desk 1 are monotonically growing features of frequency. Nonetheless, this isn’t a crucial situation for outlining this group.

1.4. Similarity encoders [5, 8, 18] rework categorical knowledge into numerical kind by making use of similarity or distance metrics that seize similarities between completely different classes.

One group of similarity encoders [8, 18] relies on a morphological comparability between two classes handled as strings. Examples of similarity metrics are Levenshtein’s ratio, Jaro-Winkler similarity, or N-gram similarity. The specific variable is then encoded as a vector, the place every dimension corresponds to a pairwise comparability of a reference class with all classes, and the worth represents the computed similarity rating (just like establishing a variance-covariance matrix). Encoders of this group sometimes create ok new options. This encoding is especially helpful for dealing with “soiled” categorical datasets which will include typos and redundancies [18]. One can consider One-Scorching encoding as a particular case of similarity encoding, the place the similarity measure can take solely two values, 0 or 1.

One other group of similarity encoders [5], together with, e.g. Embeddings, Min-Hash, or Gamma-Poisson matrix factorization, is designed for top cardinality classes (ok>>1). They challenge categorical options right into a decrease dimensional house. This group is thus just like binning encoders, that are additionally significantly helpful for giant ok however don’t intention to protect the morphological similarity of classes.

2. Supervised encoders use details about the goal variable. For regression or binary classification duties, they sometimes create a single new function for every categorical one.

2.1. Easy goal encoders seize the connection between the explicit attribute and the goal. Establishing a easy goal encoder entails calculating a statistic primarily based on the goal variable for every degree of the explicit variable. Frequent statistics embrace the imply, median, or likelihood ratios of the goal variable conditional on every class.

The process is as follows:

• For every class degree, group the info by the explicit variable.
• Inside every group, compute the specified statistic of curiosity.
• For every occasion within the knowledge, substitute the unique categorical worth with the corresponding statistic.

Easy goal encoding runs the danger of overfitting, particularly for small knowledge units and classes with only a few observations. Methods similar to smoothing (regularization) and cross-validation, which we describe shortly, may help mitigate this threat.

2.2. Smoothing encoders are generalizations of straightforward goal encoders that introduce a smoothing parameter. The aim of this smoothing parameter is to forestall overfitting and to enhance the generalization of the encoder to new knowledge, particularly when there are classes with a small variety of observations. A typical system for the smoothed worth is

the place m is the variety of instances the class happens within the knowledge set. It could barely differ, see [13]. By adjusting the smoothing parameter, you possibly can management the steadiness between the class statistic and the general statistic. A bigger smoothing parameter makes the encoding much less delicate to the category-specific goal statistic. Setting the smoothing parameter to zero within the above system leads to a easy goal encoding.

2.3. Information-constraining encoders use info from a subset of rows in a knowledge set. An instance of a data-constraining coder is the leave-one-out coder, which, for a given categorical worth, calculates the statistic of the goal variable for all different situations by which the class happens, excluding the present occasion. This method ensures that the encoded worth for a given knowledge set doesn’t embrace its personal goal worth.

Information-constraining methods assist create encodings which are extra consultant of the true relationships within the unseen knowledge and fewer liable to overfitting. The variety of distinctive values within the encoded column could exceed the cardinality of the unique categorical column ok. One can even introduce smoothing into data-constraining encodings.

3. Binning encoders may be each supervised and unsupervised. Usually, their work may be divided into two steps. Step one creates an auxiliary categorical function by sorting the unique classes into bins. The second step applies an encoder from one of many different teams mentioned above to this auxiliary function. Each steps may be both unsupervised or supervised independently of one another. For instance, in step one, you possibly can kind the bins by grouping uncommon classes collectively (unsupervised, e.g., One-Scorching MC encoder); alternatively, you possibly can apply a easy goal encoder and group the classes with shut encoded values collectively (supervised, e.g., Discretized Goal encoder).

Some binning encoders, e.g. Hashing, should not have this two-stage construction. The variety of new options created by binning encoders is normally <ok.

Tuning methods

Earlier than continuing with the outcomes, I’ll briefly summarize the tuning methods from our benchmark. That is vital to grasp the scope of the next figures and to reuse the info and code from our benchmark. The three tuning methods are

No tuning:

• Encode the explicit columns of the coaching knowledge;
• Practice an ML mannequin with its default hyperparameters.

Mannequin tuning:

• Encode the explicit columns of the coaching knowledge;
• Divide the coaching knowledge into folds;
• Optimize the hyperparameters of an ML mannequin utilizing cross-validation.

Full tuning:

• Cut up coaching knowledge into folds;
• Encode every coaching fold individually (in case of information constraining encoder household, every fold is additional cut up into the nested folds);
• Optimize the hyperparameters of an ML mannequin with cross-validation.

Outcomes

I’ll title the profitable encoders from our experiments, level out which tuning technique carried out finest and must be a part of your ML pipeline, and current the runtimes for numerous encoders.

Rating

In Determine 1, the boxplots present the distribution of encoder ranks throughout all datasets, high quality metrics, and tuning methods. That’s, every boxplot contains ~50×4×3=600 experiments for particular person fashions and ~50×4×3×5=3000 experiments for all fashions, excluding the small fraction of experiments that timed out or failed for different causes. We noticed that 4 encoders: One-Scorching, Binary (‘Bin’ on the plot), Sum, and Weight of Proof are constantly among the many finest performing. For logistic regression, the distinction of those 4 from the remainder is statistically important. This result’s considerably shocking since many earlier research (see [13, 14]) have reported drawbacks of unsupervised encoders, particularly One-Scorching. The precise meanings of all different encoder abbreviations on Determine 1 will not be vital for the remainder, however yow will discover them in [2].

Tuning technique

Determine 2 plots the efficiency distinction between full and no tuning methods (as I described above) for every categorical encoder. Every covers the experiments with completely different datasets, ML fashions, and efficiency metrics. Determine 3 is an analogous plot evaluating full and mannequin tuning methods. Be aware that these plots don’t embrace a number of the ML fashions and datasets as a result of we restricted the computational complexity. See [2] for particulars and extra plots.

Based mostly on these outcomes, I might usually suggest sticking to the total tuning technique, i.e., encoding knowledge for every fold individually when optimizing the hyperparameters of an ML mannequin; that is particularly vital for knowledge constraining encoders.

Runtime

Lastly, Determine 4 plots the instances required to encode knowledge units on a logarithmic scale. Easy goal and binning encoders are the quickest, whereas smoothing and knowledge constraining encoders are the slowest. The 4 finest performing encoders — One-Scorching, Binary (‘Bin’ on the plot), Sum, and Weight of Proof — take an inexpensive period of time to course of knowledge.

Reusing the code and the outcomes

A pleasant function of our benchmark is which you can simply prolong it to check your method to categorical encoding and put it within the context of present experimental outcomes. For instance, chances are you’ll wish to take a look at the encoders from Desk 1 that aren’t a part of our benchmark, or chances are you’ll wish to take a look at your customized composite encoder like if n/ok>a then One-Scorching MC else One-Scorching the place ok is the function cardinality and n is the dimensions of the dataset, as earlier than. We clarify the method on GitHub and share the demonstration of how to do that within the kaggle pocket book [17]. Beneath is a brief abstract of [17].

For illustrative functions, we restricted the instance to a logistic regression mannequin with out hyperparameter tuning. The corresponding outcomes of our benchmark are proven in Determine 5 under.

Now suppose that Yury Kashnitsky, who kindly offered suggestions on our paper, wonders whether or not the hashing encoder is aggressive with the encoders we evaluated. In [17], we present how one can carry out the lacking experiments and discover that the hashing encoder performs moderately effectively with our alternative of its hyperparameter, as Determine 6 exhibits.

Conclusion

I summarized our benchmark of categorical encoders and defined how one can profit from our shared artifacts. You could have discovered:

• Categorical encoders, as ML fashions, may be supervised or unsupervised.
• I offered eight households of encoders. Some encoders protect the cardinality of categorical options, others can cut back it (usually the case for binning, but additionally for another encoders, e.g., Naive Goal) or enhance it (knowledge constraining).
• One-Scorching, Binary, Sum, and Weight of Proof encoders carried out finest on common in our experiments, particularly with logistic regression.
• See the kaggle pocket book [17] for a demo of how one can add the specified encoders to the benchmark and plot the outcomes; the required code is accessible on GitHub.

Acknowledgements

This story wouldn’t exist if Federico Matteucci, my colleague and first writer in [2], had not written the code for the benchmark and for the kaggle pocket book [17].

Assets

[1] UC Irvine Machine Studying Repository
[2] Matteucci, F., Arzamasov, V., & Böhm, Okay. (2024). A benchmark of categorical encoders for binary classification. Advances in Neural Info Processing Methods, 36. (paper, code on GitHub, poster)
[3] Seca, D., & Mendes-Moreira, J. (2021). Benchmark of encoders of nominal options for regression. In World Convention on Info Methods and Applied sciences (pp. 146–155). Cham: Springer Worldwide Publishing.
[4] Pargent, F., Pfisterer, F., Thomas, J., & Bischl, B. (2022). Regularized goal encoding outperforms conventional strategies in supervised machine studying with excessive cardinality options. Computational Statistics, 37(5), 2671–2692.
[5] Cerda, P., & Varoquaux, G. (2020). Encoding high-cardinality string categorical variables. IEEE Transactions on Information and Information Engineering, 34(3), 1164–1176.
[6] Potdar, Okay., Pardawala, T. S., & Pai, C. D. (2017). A comparative research of categorical variable encoding methods for neural community classifiers. Worldwide journal of pc purposes, 175(4), 7–9.
[7] Dahouda, M. Okay., & Joe, I. (2021). A deep-learned embedding approach for categorical options encoding. IEEE Entry, 9, 114381–114391.
[8] Cerda, P., Varoquaux, G., & Kégl, B. (2018). Similarity encoding for studying with soiled categorical variables. Machine Studying, 107(8), 1477–1494.
[9] Wright, M. N., & König, I. R. (2019). Splitting on categorical predictors in random forests. PeerJ, 7, e6339.
[10] Gnat, S. (2021). Affect of categorical variables encoding on property mass valuation. Procedia Laptop Science, 192, 3542–3550.
[11] Johnson, J. M., & Khoshgoftaar, T. M. (2021, August). Encoding methods for high-cardinality options and ensemble learners. In 2021 IEEE twenty second worldwide convention on info reuse and integration for knowledge science (IRI) (pp. 355–361). IEEE.
[12] Valdez-Valenzuela, E., Kuri-Morales, A., & Gomez-Adorno, H. (2021). Measuring the impact of categorical encoders in machine studying duties utilizing artificial knowledge. In Advances in Computational Intelligence: twentieth Mexican Worldwide Convention on Synthetic Intelligence, MICAI 2021, Mexico Metropolis, Mexico, October 25–30, 2021, Proceedings, Half I 20 (pp. 92–107). Springer Worldwide Publishing.
[13] Benchmarking Categorical Encoders (a narrative on TDS)
[14] Categorical Encoding: Key Insights (my story on Medium)
[15] R Library Distinction Coding Methods for categorical variables
[16] Class Encoders (python module)
[17] Add a customized encoder to the benchmark (kaggle pocket book)
[18] Similarity Encoding for Soiled Classes Utilizing dirty_cat (a narrative on TDS)