Conventional Method

Many current implementations on survival evaluation begin off with a dataset containing one remark per particular person (sufferers in a well being examine, workers within the attrition case, purchasers within the shopper churn case, and so forth). For these people we sometimes have two key variables: one signaling the occasion of curiosity (an worker quitting) and one other measuring time (how lengthy they’ve been with the corporate, as much as both at this time or their departure). Along with these two variables, we then have explanatory variables with which we purpose to foretell the danger of every particular person. These options can embrace the job function, age or compensation of the worker, for instance.

Shifting on, most implementations on the market take a survival mannequin (from easier estimators equivalent to Kaplan Meier to extra advanced ones like ensemble fashions and even neural networks), match them over a prepare set after which consider over a take a look at set. This train-test break up is normally carried out over the person observations, usually making a stratified break up.

In my case, I began with a dataset that adopted a number of workers in an organization month-to-month till December 2023 (in case the worker was nonetheless on the firm), or till the month they left the corporate — the occasion date:

As a way to adapt my information to the survival case, I took the final remark of every worker as proven within the image above (the blue dots for lively workers, and the purple crosses for workers who left). At that time for every worker, I recorded whether or not the occasion had occurred at that date or not (in the event that they have been lively or if they’d left), their tenure in months at the moment, and all their explanatory variables. I then carried out a stratified train-test break up over this information, like this:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
# We load our dataset with a number of observations (record_date) per worker (employee_id)
# The occasion column signifies if the worker left on that given month (1) or if the worker was nonetheless lively (0)
df = pd.read_csv(f'{FILE_NAME}.csv')
# Making a label the place constructive occasions have tenure and unfavourable occasions have unfavourable tenure - required by Random Survival Forest
df_model['label'] = np.the place(df_model['event'], df_model['tenure_in_months'], - df_model['tenure_in_months'])
df_train, df_test = train_test_split(df_model, test_size=0.2, stratify=df_model['event'], random_state=42)

After performing the break up, I proceeded to suit a mannequin. On this case, I selected to experiment with a Random Survival Forest utilizing the scikit-survival library.

from sklearn.preprocessing import OrdinalEncoder
from sksurv.datasets import get_x_y
from sksurv.ensemble import RandomSurvivalForest
cat_features = [] # listing of all the explicit options
options = [] # listing of all of the options (each categorical and numeric)
# Categorical Encoding
encoder = OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)
encoder.match(df_train[cat_features])
df_train[cat_features] = encoder.remodel(df_train[cat_features])
df_test[cat_features] = encoder.remodel(df_test[cat_features])
# X & y
X_train, y_train = get_x_y(df_train, attr_labels=['event','tenure_in_months'], pos_label=1)
X_test, y_test = get_x_y(df_test, attr_labels=['event','tenure_in_months'], pos_label=1)
# Match the mannequin
estimator = RandomSurvivalForest(random_state=RANDOM_STATE)
estimator.match(X_train[features], y_train)
# Retailer predictions
y_pred = estimator.predict(X_test[features])

After a fast run utilizing the default settings of the mannequin, I used to be thrilled with the take a look at metrics I noticed. To start with, I used to be getting a concordance index above 0.90 within the take a look at set. The concordance index is a measure of how effectively the mannequin predicts the order of occasions: it displays whether or not workers predicted to be at excessive threat have been certainly those leaving the corporate first. An index of 1 corresponds to good prediction accuracy, whereas an index of 0.5 signifies a prediction no higher than random likelihood.

I used to be notably concerned with seeing if the staff who left within the take a look at set matched with probably the most dangerous workers in line with the mannequin. Within the case of the Random Survival Forest, the mannequin returns the danger scores of every remark. I took the share of workers who left the corporate within the take a look at set, and used it to filter probably the most dangerous workers in line with the mannequin. The outcomes have been very strong, with the staff flagged with probably the most threat matching virtually completely with the precise leavers, with an F1 rating above 0.90 within the minority class.

from lifelines.utils import concordance_index
from sklearn.metrics import classification_report
# Concordance Index
ci_test = concordance_index(df_test['tenure_in_months'], -y_pred, df_test['event'])
print(f'Concordance index:{ci_test:0.5f}n')
# Match probably the most dangerous workers (in line with the mannequin) with the staff who left
q_test = 1 - df_test['event'].imply()
thr = np.quantile(y_pred, q_test)
risky_employees = (y_pred >= thr) * 1
print(classification_report(df_test['event'], risky_employees))

Getting +0.9 metrics on the primary run ought to set off an alarm: was the mannequin actually capable of predict whether or not an worker was going to remain or go away with such confidence? Think about this: we submit our predictions saying which workers are more than likely to depart. Nonetheless, a pair months go by, and HR then reaches us fearful, saying that the individuals who left over the last interval, didn’t precisely match with our predictions, not less than on the charge it was anticipated from our take a look at metrics.

We have now two major issues right here: the primary one is that our mannequin isn’t extrapolating fairly in addition to we thought. The second, and even worse, is that we weren’t capable of measure this lack of efficiency. First, I’ll present a easy manner we are able to estimate how effectively our mannequin is actually extrapolating, after which I’ll discuss one potential motive it might be failing to take action, and easy methods to mitigate it.

Estimating Generalization Capabilities

The important thing right here is getting access to panel information, that’s, a number of data of our people over time, up till the time of occasion or the time the examine ended (the date of our snapshot, within the case of worker attrition). As an alternative of discarding all this data and preserving solely the final file of every worker, we may use it to create a take a look at set that may higher replicate how the mannequin performs sooner or later. The concept is kind of easy: suppose we have now month-to-month data of our workers up till December 2023. We may transfer again, say, 6 months, and faux we took the snapshot in June as an alternative of December. Then, we’d take the final remark for workers who left the corporate earlier than June 2023 as constructive occasions, and the June 2023 file of workers who survived past that date as unfavourable occasions, even when we already know a few of them finally left afterwards. We’re pretending we don’t know this but.

As the image above exhibits, I take a snapshot in June, and all workers who have been lively at the moment are taken as lively. The take a look at dataset takes all these lively workers at June with their explanatory variables as they have been on that date, and takes the newest tenure they achieved by December:

test_date = '2023-07-01'
# Choosing coaching information from data earlier than the take a look at date and taking the final remark per worker
df_train = df[df.record_date < test_date].reset_index(drop=True).copy()
df_train = df_train.groupby('employee_id').tail(1).reset_index(drop=True)
df_train['label'] = np.the place(df_train['event'], df_train['tenure_in_months'], - df_train['tenure_in_months'])
# Getting ready take a look at information with data of lively workers on the take a look at date
df_test = df[(df.record_date == test_date) & (df['event']==0)].reset_index(drop=True).copy()
df_test = df_test.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.drop(columns = ['tenure_in_months','event'])
# Fetching the final tenure and occasion standing for workers within the take a look at dataset
df_last_tenure = df[df.employee_id.isin(df_test.employee_id.unique())].reset_index(drop=True).copy()
df_last_tenure = df_last_tenure.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.merge(df_last_tenure[['employee_id','tenure_in_months','event']], how='left')
df_test['label'] = np.the place(df_test['event'], df_test['tenure_in_months'], - df_test['tenure_in_months'])

We match our mannequin once more on this new prepare information, and as soon as we end we make our predictions for all workers who have been lively on June. We then evaluate these predictions to the precise consequence of July — December 2023 — that is our take a look at set. If these workers we marked as having probably the most threat left in the course of the semester, and people we marked as having the bottom threat didn’t go away, or left fairly late within the interval, then our mannequin is extrapolating effectively. By shifting our evaluation again in time and leaving the final interval for analysis, we are able to have a greater understanding of how effectively our mannequin is generalizing. After all, we may take this one step additional and carry out some sort of time-series cross validation. For instance, we may iterate this course of many instances, every time transferring 6 months again in time, and evaluating the mannequin’s accuracy over a number of time frames.

After coaching our mannequin as soon as once more, we now see a drastic lower in efficiency. To start with, the concordance index is now round 0.5 — equal to that of a random predictor. Additionally, if we attempt to match the ‘n’ most dangerous workers in line with the mannequin with the ‘n’ workers who left within the take a look at set, we see a really poor classification with a 0.15 F1 for the minority class:

So clearly there’s something unsuitable, however not less than we are actually capable of detect it as an alternative of being misled. The principle takeaway right here is that our mannequin performs effectively with a conventional break up, however doesn’t extrapolate when doing a time-based break up. This can be a clear signal that a while bias could also be current. Briefly, time-dependent data is being leaked and our mannequin is overfitting over it. That is frequent in circumstances like our worker attrition downside, when the dataset comes from a snapshot taken at some date.

Time Bias

The issue cuts right down to this: all our constructive observations (workers who left) belong to previous dates, and all our unfavourable observations (presently lively workers) are all measured on the identical date — at this time. If there’s a single function that reveals this to the mannequin, then as an alternative of predicting threat we can be predicting if an worker was recorded in December 2023 or earlier than. This could possibly be very delicate. For instance, one function we could possibly be utilizing is the engagement rating of the staff. This function may effectively present some seasonal patterns, and measuring it on the similar time for lively workers will certainly introduce some bias within the mannequin. Possibly in December, in the course of the vacation season, this engagement rating tends to lower. The mannequin will see a low rating related to all lively workers, so it might study to foretell that at any time when the engagement runs low, the churn threat additionally goes down, when in reality it ought to be the alternative!

By now, a easy but fairly efficient answer for this downside ought to be clear: as an alternative of taking the final remark for every lively worker, we may simply decide a random month from all their historical past throughout the firm. This may strongly scale back the probabilities of the mannequin choosing on any temporal patterns that we don’t want it to overfit on:

Within the image above we are able to see that we are actually spanning a broader set of dates for the lively workers. As an alternative of utilizing their blue dots at June 2023, we take the random orange dots as an alternative, and file their variables on the time, and the tenure they’d up to now within the firm:

np.random.seed(0)
# Choose coaching information earlier than the take a look at date
df_train = df[df.record_date < test_date].reset_index(drop=True).copy()
# Create an indicator for whether or not an worker finally churns throughout the prepare set
df_train['indicator'] = df_train.groupby('employee_id').occasion.remodel(max)
# Isolate data of workers who left, and retailer their final remark
churn = df_train[df_train.indicator==1].reset_index(drop=True).copy()
churn = churn.groupby('employee_id').tail(1).reset_index(drop=True)
# For workers who stayed, randomly decide one remark from their historic data
keep = df_train[df_train.indicator==0].reset_index(drop=True).copy()
keep = keep.groupby('employee_id').apply(lambda x: x.pattern(1)).reset_index(drop=True)
# Mix churn and keep samples into the brand new coaching dataset
df_train = pd.concat([churn,stay], ignore_index=True).copy()
df_train['label'] = np.the place(df_train['event'], df_train['tenure_in_months'], - df_train['tenure_in_months'])
del df_train['indicator']
# Put together the take a look at dataset equally, utilizing solely the snapshot from the take a look at date
df_test = df[(df.record_date == test_date) & (df.event==0)].reset_index(drop=True).copy()
df_test = df_test.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.drop(columns = ['tenure_in_months','event'])
# Get the final recognized tenure and occasion standing for workers within the take a look at set
df_last_tenure = df[df.employee_id.isin(df_test.employee_id.unique())].reset_index(drop=True).copy()
df_last_tenure = df_last_tenure.groupby('employee_id').tail(1).reset_index(drop=True)
df_test = df_test.merge(df_last_tenure[['employee_id','tenure_in_months','event']], how='left')
df_test['label'] = np.the place(df_test['event'], df_test['tenure_in_months'], - df_test['tenure_in_months'])

We then prepare our mannequin as soon as once more, and consider it over the identical take a look at set we had earlier than. We now see a concordance index of round 0.80. This isn’t the +0.90 we had earlier, but it surely positively is a step up from the random-chance stage of 0.5. Concerning our curiosity in classifying workers, we’re nonetheless very far off the +0.9 F1 we had earlier than, however we do see a slight enhance in comparison with the earlier strategy, particularly for the minority class.