Featurizing time sequence information into a normal tabular format for classical ML fashions and enhancing accuracy utilizing AutoML

This text delves into enhancing the method of forecasting every day vitality consumption ranges by reworking a time sequence dataset right into a tabular format utilizing open-source libraries. We discover the applying of a well-liked multiclass classification mannequin and leverage AutoML with Cleanlab Studio to considerably increase our out-of-sample accuracy.

The important thing takeaway from this text is that we are able to make the most of extra basic strategies to mannequin a time sequence dataset by changing it to a tabular construction, and even discover enhancements in attempting to foretell this time sequence information.

At a excessive stage we’ll:

• Set up a baseline accuracy by becoming a Prophet forecasting mannequin on our time sequence information
• Convert our time sequence information right into a tabular format through the use of open-source featurization libraries after which will present that may outperform our Prophet mannequin with a normal multiclass classification (Gradient Boosting) strategy by a 67% discount in prediction error (enhance by 38% uncooked proportion factors in out-of-sample accuracy).
• Use an AutoML answer for multiclass classification resulted in a 42% discount in prediction error (enhance by 8% in uncooked proportion factors in out-of-sample accuracy) in comparison with our Gradient Boosting mannequin and resulted in a 81% discount in prediction error (enhance by 46% in uncooked proportion factors in out-of-sample accuracy) in comparison with our Prophet forecasting mannequin.

To run the code demonstrated on this article, right here’s the full pocket book.

You may obtain the dataset right here.

The information represents PJM hourly vitality consumption (in megawatts) on an hourly foundation. PJM Interconnection LLC (PJM) is a regional transmission group (RTO) in the USA. It’s a part of the Jap Interconnection grid working an electrical transmission system serving many states.

Let’s check out our dataset. The information consists of one datetime column (object kind), and the Megawatt Power Consumption (float64) kind) column we try to forecast as a discrete variable (akin to the quartile of hourly vitality consumption ranges). Our goal is to coach a time sequence forecasting mannequin to have the ability to forecast the tomorrow’s every day vitality consumption stage falling into 1 of 4 ranges: low , under common , above common or excessive (these ranges had been decided based mostly on quartiles of the general every day consumption distribution). We first show easy methods to apply time-series forecasting strategies like Prophet to this drawback, however these are restricted to sure forms of ML fashions appropriate for time-series information. Subsequent we show easy methods to reframe this drawback into a normal multiclass classification drawback that we are able to apply any machine studying mannequin to, and present how we are able to get hold of superior forecasts through the use of highly effective supervised ML.

We first convert this information right into a common vitality consumption at a every day stage and rename the columns to the format that the Prophet forecasting mannequin expects. These real-valued every day vitality consumption ranges are transformed into quartiles, which is the worth we try to foretell. Our coaching information is proven under together with the quartile every every day vitality consumption stage falls into. The quartiles are computed utilizing coaching information to forestall information leakage.

We then present the check information under, which is the information we’re evaluating our forecasting outcomes in opposition to.

We then present the check information under, which is the information we’re evaluating our forecasting outcomes in opposition to.

As seen within the pictures above, we’ll use a date cutoff of 2015-04-09 to finish the vary of our coaching information and begin our check information at 2015-04-10 . We compute quartile thresholds of our every day vitality consumption utilizing ONLY coaching information. This avoids information leakage – utilizing out-of-sample information that’s obtainable solely sooner or later.

Subsequent, we’ll forecast the every day PJME vitality consumption stage (in MW) at some point of our check information and characterize the forecasted values as a discrete variable. This variable represents which quartile the every day vitality consumption stage falls into, represented categorically as 1 (low), 2 (under common), 3 (above common), or 4 (excessive). For analysis, we’re going to use the accuracy_score perform from scikit-learn to judge the efficiency of our fashions. Since we’re formulating the issue this manner, we’re in a position to consider our mannequin’s next-day forecasts (and examine future fashions) utilizing classification accuracy.

import numpy as np
from prophet import Prophet
from sklearn.metrics import accuracy_score
# Initialize mannequin and prepare it on coaching information
mannequin = Prophet()
mannequin.match(train_df)
# Create a dataframe for future predictions masking the check interval
future = mannequin.make_future_dataframe(durations=len(test_df), freq='D')
forecast = mannequin.predict(future)
# Categorize forecasted every day values into quartiles based mostly on the thresholds
forecast['quartile'] = pd.minimize(forecast['yhat'], bins = [-np.inf] + checklist(quartiles) + [np.inf], labels=[1, 2, 3, 4])
# Extract the forecasted quartiles for the check interval
forecasted_quartiles = forecast.iloc[-len(test_df):]['quartile'].astype(int)
# Categorize precise every day values within the check set into quartiles
test_df['quartile'] = pd.minimize(test_df['y'], bins=[-np.inf] + checklist(quartiles) + [np.inf], labels=[1, 2, 3, 4])
actual_test_quartiles = test_df['quartile'].astype(int)
# Calculate the analysis metrics
accuracy = accuracy_score(actual_test_quartiles, forecasted_quartiles)
# Print the analysis metrics
print(f'Accuracy: {accuracy:.4f}')
>>> 0.4249

The out-of-sample accuracy is kind of poor at 43%. By modelling our time sequence this manner, we restrict ourselves to solely use time sequence forecasting fashions (a restricted subset of doable ML fashions). Within the subsequent part, we take into account how we are able to extra flexibly mannequin this information by reworking the time-series into a normal tabular dataset by way of applicable featurization. As soon as the time-series has been reworked into a normal tabular dataset, we’re in a position to make use of any supervised ML mannequin for forecasting this every day vitality consumption information.

Now we convert the time sequence information right into a tabular format and featurize the information utilizing the open supply libraries sktime, tsfresh, and tsfel. By using libraries like these, we are able to extract a wide selection of options that seize underlying patterns and traits of the time sequence information. This consists of statistical, temporal, and presumably spectral options, which give a complete snapshot of the information’s habits over time. By breaking down time sequence into particular person options, it turns into simpler to know how totally different facets of the information affect the goal variable.

TSFreshFeatureExtractor is a function extraction device from the sktime library that leverages the capabilities of tsfresh to extract related options from time sequence information. tsfresh is designed to routinely calculate an enormous variety of time sequence traits, which could be extremely helpful for understanding advanced temporal dynamics. For our use case, we make use of the minimal and important set of options from our TSFreshFeatureExtractor to featurize our information.

tsfel, or Time Collection Function Extraction Library, provides a complete suite of instruments for extracting options from time sequence information. We make use of a predefined config that permits for a wealthy set of options (e.g., statistical, temporal, spectral) to be constructed from the vitality consumption time sequence information, capturing a variety of traits that is likely to be related for our classification activity.

import tsfel
from sktime.transformations.panel.tsfresh import TSFreshFeatureExtractor
# Outline tsfresh function extractor
tsfresh_trafo = TSFreshFeatureExtractor(default_fc_parameters="minimal")
# Remodel the coaching information utilizing the function extractor
X_train_transformed = tsfresh_trafo.fit_transform(X_train)
# Remodel the check information utilizing the identical function extractor
X_test_transformed = tsfresh_trafo.remodel(X_test)
# Retrieves a pre-defined function configuration file to extract all obtainable options
cfg = tsfel.get_features_by_domain()
# Perform to compute tsfel options per day
def compute_features(group):
# TSFEL expects a DataFrame with the information in columns, so we transpose the enter group
options = tsfel.time_series_features_extractor(cfg, group, fs=1, verbose=0)
return options
# Group by the 'day' stage of the index and apply the function computation
train_features_per_day = X_train.groupby(stage='Date').apply(compute_features).reset_index(drop=True)
test_features_per_day = X_test.groupby(stage='Date').apply(compute_features).reset_index(drop=True)
# Mix every featurization right into a set of mixed options for our prepare/check information
train_combined_df = pd.concat([X_train_transformed, train_features_per_day], axis=1)
test_combined_df = pd.concat([X_test_transformed, test_features_per_day], axis=1)

Subsequent, we clear our dataset by eradicating options that confirmed a excessive correlation (above 0.8) with our goal variable — common every day vitality consumption ranges — and people with null correlations. Excessive correlation options can result in overfitting, the place the mannequin performs effectively on coaching information however poorly on unseen information. Null-correlated options, however, present no worth as they lack a definable relationship with the goal.

By excluding these options, we goal to enhance mannequin generalizability and be certain that our predictions are based mostly on a balanced and significant set of information inputs.

# Filter out options which can be extremely correlated with our goal variable
column_of_interest = "PJME_MW__mean"
train_corr_matrix = train_combined_df.corr()
train_corr_with_interest = train_corr_matrix[column_of_interest]
null_corrs = pd.Collection(train_corr_with_interest.isnull())
false_features = null_corrs[null_corrs].index.tolist()
columns_to_exclude = checklist(set(train_corr_with_interest[abs(train_corr_with_interest) > 0.8].index.tolist() + false_features))
columns_to_exclude.take away(column_of_interest)
# Filtered DataFrame excluding columns with excessive correlation to the column of curiosity
X_train_transformed = train_combined_df.drop(columns=columns_to_exclude)
X_test_transformed = test_combined_df.drop(columns=columns_to_exclude)

If we have a look at the primary a number of rows of the coaching information now, this can be a snapshot of what it seems like. We now have 73 options that had been added from the time sequence featurization libraries we used. The label we’re going to predict based mostly on these options is the following day’s vitality consumption stage.

It’s necessary to notice that we used a greatest apply of making use of the featurization course of individually for coaching and check information to keep away from information leakage (and the held-out check information are our most up-to-date observations).

Additionally, we compute our discrete quartile worth (utilizing the quartiles we initially outlined) utilizing the next code to acquire our prepare/check vitality labels, which is what our y_labels are.

# Outline a perform to categorise every worth right into a quartile
def classify_into_quartile(worth):
if worth < quartiles[0]:
return 1  
elif worth < quartiles[1]:
return 2  
elif worth < quartiles[2]:
return 3  
else:
return 4  
y_train = X_train_transformed["PJME_MW__mean"].rename("daily_energy_level")
X_train_transformed.drop("PJME_MW__mean", inplace=True, axis=1)
y_test = X_test_transformed["PJME_MW__mean"].rename("daily_energy_level")
X_test_transformed.drop("PJME_MW__mean", inplace=True, axis=1)
energy_levels_train = y_train.apply(classify_into_quartile)
energy_levels_test = y_test.apply(classify_into_quartile)

Utilizing our featurized tabular dataset, we are able to apply any supervised ML mannequin to foretell future vitality consumption ranges. Right here we’ll use a Gradient Boosting Classifier (GBC) mannequin, the weapon of selection for many information scientists working on tabular information.

Our GBC mannequin is instantiated from the sklearn.ensemble module and configured with particular hyperparameters to optimize its efficiency and keep away from overfitting.

from sklearn.ensemble import GradientBoostingClassifier
gbc = GradientBoostingClassifier(
n_estimators=150,
learning_rate=0.1,
max_depth=4,
min_samples_leaf=20,
max_features='sqrt',
subsample=0.8,
random_state=42
)
gbc.match(X_train_transformed, energy_levels_train)
y_pred_gbc = gbc.predict(X_test_transformed)
gbc_accuracy = accuracy_score(energy_levels_test, y_pred_gbc)
print(f'Accuracy: {gbc_accuracy:.4f}')
>>> 0.8075

The out-of-sample accuracy of 81% is significantly higher than our prior Prophet mannequin outcomes.

Now that we’ve seen easy methods to featurize the time-series drawback and the advantages of making use of highly effective ML fashions like Gradient Boosting, a pure query emerges: Which supervised ML mannequin ought to we apply? After all, we may experiment with many fashions, tune their hyperparameters, and ensemble them collectively. A neater answer is to let AutoML deal with all of this for us.

Right here we’ll use a easy AutoML answer offered in Cleanlab Studio, which entails zero configuration. We simply present our tabular dataset, and the platform routinely trains many forms of supervised ML fashions (together with Gradient Boosting amongst others), tunes their hyperparameters, and determines which fashions are greatest to mix right into a single predictor. Right here’s all of the code wanted to coach and deploy an AutoML supervised classifier:

from cleanlab_studio import Studio
studio = Studio()
studio.create_project(
dataset_id=energy_forecasting_dataset,
project_name="ENERGY-LEVEL-FORECASTING",
modality="tabular",
task_type="multi-class",
model_type="common",
label_column="daily_energy_level",
)
mannequin = studio.get_model(energy_forecasting_model)
y_pred_automl = mannequin.predict(test_data, return_pred_proba=True)

Beneath we are able to see mannequin analysis estimates within the AutoML platform, exhibiting all the several types of ML fashions that had been routinely match and evaluated (together with a number of Gradient Boosting fashions), in addition to an ensemble predictor constructed by optimally combining their predictions.

After working inference on our check information to acquire the next-day vitality consumption stage predictions, we see the check accuracy is 89%, a 8% uncooked proportion factors enchancment in comparison with our earlier Gradient Boosting strategy.

For our PJM every day vitality consumption information, we discovered that remodeling the information right into a tabular format and featurizing it achieved a 67% discount in prediction error (enhance by 38% in uncooked proportion factors in out-of-sample accuracy) in comparison with our baseline accuracy established with our Prophet forecasting mannequin.

We additionally tried a straightforward AutoML strategy for multiclass classification, which resulted in a 42% discount in prediction error (enhance by 8% in uncooked proportion factors in out-of-sample accuracy) in comparison with our Gradient Boosting mannequin and resulted in a 81% discount in prediction error (enhance by 46% in uncooked proportion factors in out-of-sample accuracy) in comparison with our Prophet forecasting mannequin.

By taking approaches like these illustrated above to mannequin a time sequence dataset past the constrained strategy of solely contemplating forecasting strategies, we are able to apply extra basic supervised ML methods and obtain higher outcomes for sure forms of forecasting issues.

Until in any other case famous, all pictures are by the creator.