[ad_1]
Do away with your ineffective code for filtering time collection information
Each time I work with time collection information, I find yourself writing complicated and non-reusable code to filter it. Whether or not I’m doing easy filtering methods like eradicating weekends, or extra complicated ones like eradicating particular time home windows, I all the time resort to writing a fast and soiled operate that works for the precise factor that I’m filtering within the second, however by no means once more.
I lastly determined to interrupt that horrible cycle by writing a processor that enables me to filter time collection irrespective of how complicated the situation utilizing quite simple and concise inputs.
Simply an instance of the way it works in apply:
- On weekdays, I wish to take away < 6 am and ≥ 8 pm, and on weekends I wish to take away < 8 am and ≥ 10 pm
df = pl.DataFrame(
{"date": [
# -- may 24th is a Friday, weekday
'2024-05-24 00:00:00', # < 6 am, should remove
'2024-05-24 06:00:00', # not < 6 am, should keep
'2024-05-24 06:30:00', # not < 6 am, should keep
'2024-05-24 20:00:00', # >= 8 pm, should remove
# -- may 25th is a Saturday, weekend
'2024-05-25 00:00:00', # < 8 am, should remove
'2024-05-25 06:00:00', # < 8 am, should remove
'2024-05-25 06:30:00', # < 8 am, should remove
'2024-05-25 20:00:00', # not >= 10 pm, should keep
]
}
).with_columns(pl.col("date").str.strptime(pl.Datetime, "%Y-%m-%d %H:%M:%S"))
- With out processor: expressive, however verbose and non-reusable
df.filter(
pl.Expr.not_(
(
(pl.col("date").dt.weekday() < 6)
.and_(
(pl.col("date").dt.hour() < 6)
.or_(pl.col("date").dt.hour() >= 20)
)
)
.or_(
(pl.col("date").dt.weekday() >= 6)
.and_(
(pl.col("date").dt.hour() < 8)
.or_(pl.col("date").dt.hour() >= 22)
)
)
)
)
- With processor: equally as expressive, concise and reusable
processor = FilterDataBasedOnTime(
"date", time_patterns=[
"<6wd<6h",
"<6wd>=20h",
">=6wd<8h",
">=6wd>=22h",
]
)
processor.remodel(df)
On this article I’ll clarify how I got here up with this answer, beginning with the string format I selected to outline filter situations, adopted by a design of the processor itself. In the direction of the top of the article, I’ll describe how this pipeline can be utilized alongside different pipelines to allow complicated time collection processing with just a few traces of code.
In the event you’re within the code solely, skip to the top of the article for a hyperlink to the repository.
Expressive, Concise and Versatile Time Situations?
This was by far the toughest a part of this job. Filtering time collection primarily based on time is conceptually straightforward, but it surely’s a lot more durable to do with code. My preliminary thought was to make use of a string sample that’s most intuitive to myself:
# -- take away values between 6 am (inclusive) and a couple of pm (unique)
sample = '>=06:00,<14:00'
Nonetheless with this, we instantly run into an issue: we lose flexibility. It is because 06:00 is ambiguous, because it may imply min:sec or hr:min . So we’d virtually all the time need to outline the date format a-priori.
This prevents us from permitting complicated filtering methods, resembling filtering a selected time ON particular days (e.g. solely take away values in [6am, 2pm) on a Saturday).
Extending my pattern to something resembling cron would not help either:
# cronlike pattern
pattern = ‘>=X-X-X 06:00:X, <X-X-X 20:00:X’
The above can help with selecting specific months or years, but doesn’t allow flexibility of things like weekdays. Further, it is not very expressive with all the X’s and it’s really verbose.
I knew that I needed a pattern that allows chaining of individual time series components or units. Effectively something that is just like an if-statement:
- IF day == Saturday
- AND time ≥ 06:00
- AND time < 14:00
So then I thought, why not use a pattern where you can add any conditions to a time-components, with the implicit assumption that they are all AND conditions?
# -- remove values in [6am, 2pm) on Saturday
pattern = 'day==6,time>=06:00,time<14:00'
Now we have a pattern that is expressive, however it can still be ambiguous, since time implicitly assumes a date fomat. So I decided to go further:
# -- remove values in [6am, 2pm) on Saturday
pattern = 'day==6,hour>=6,hour<14'
Now to make it less verbose, I borrowed the Polars duration string format (this is the equivalent of “frequency” if you are more familiar with Pandas), and viola:
# -- remove values in [6am, 2pm) on Saturday
pattern = '==6wd,>=6h,<14h'
What About Time Conditions that Need the OR Operator?
Let’s consider a different condition: to filter anything LESS than 6 am (inclusive) and > 2 pm (exclusive). A pattern like below would fail:
# -- remove values in (-inf, 6am], and (2pm, inf)
sample = '<=6h,>14h'
Since we’d learn it as: ≤ 6 am AND > 2 pm
No such worth exists that satisfies these two situations!
However the answer to that is easy: apply AND situations inside a sample, and apply OR situations throughout completely different patterns. So:
# -- take away values in (-inf, 6am], and (2pm, inf)
patterns = ['<=6h', '>14h']
Can be learn as: ≤ 6 am OR > 2pm
Why not enable OR statements inside a sample?
I did contemplate including assist for an OR assertion inside the sample, e.g. utilizing | or alternatively to let , denote the distinction between a “left” and “proper” situation. Nonetheless, I discovered that these can be including pointless complexity to parsing the sample, with out making the code any extra expressive.
I a lot want it easy: inside a sample we apply AND, throughout patterns we apply OR.
Edge Circumstances
There’s one edge-cases price discussing right here. The “if-statement” like sample doesn’t all the time work.
Let’s contemplate filtering timestamps > 06:00. If we merely outlined:
# -- sample to take away values > 06:00
sample = '>6h'
Then can we interpret this as:
- Take away all values the place hour>6
- Or take away all values the place time>06:00 ?
The latter makes extra sense, however the present sample doesn’t enable us to precise that. So to explicitly state that we which to incorporate timestamps better than the sixth hour of the day, we should add what I name the cascade operator:
# -- sample to take away values > 06:00
sample = '>6h*'
Which might be learn as:
- hour > 6
- OR (hour == 6 AND any(minute, second, millisecond, and so forth… > 0)
Which might be an correct situation to seize time>06:00!
The Code
Right here I spotlight necessary design bits to create a processor for filtering time collection information.
Parsing Logic
Because the sample is kind of easy, parsing it’s very easy. All we have to do is loop over every sample and preserve observe of the operator characters. What stays is then an inventory of operators, and an inventory of durations that they’re utilized to.
# -- code for parsing a time sample, e.g. "==7d<7h"
sample = sample.change(" ", "")
operator = ""
operators = []
duration_string = ""
duration_strings = []
for char in sample:
if char in {">", "<", "=", "!"}:
operator += char
if duration_string:
duration_strings.append(duration_string)
duration_string = ""
else:
duration_string += char
if operator:
operators.append(operator)
operator = ""
duration_strings.append(duration_string)
Now for every operator and length string, we will extract metadata that helps us make the precise boolean guidelines later on.
# -- code for extracting metadata from a parsed sample
# -- mapping to transform every operator to the Polars technique
OPERATOR_TO_POLARS_METHOD_MAPPING = {
"==": "eq",
"!=": "ne",
"<=": "le",
"<": "lt",
">": "gt",
">=": "ge",
}
operator_method = (
OPERATOR_TO_POLARS_METHOD_MAPPING[operator]
)
# -- establish cascade operations
if duration_string.endswith("*"):
duration_string = duration_string[:-1]
how = "cascade"
else:
how = "easy"
# -- extract a polars length, e.g. 7d7h into it's elements: [(7, "d"), (7, "h")]
polars_duration = PolarsDuration(length=duration_string)
decomposed_duration = polars_duration.decomposed_duration
# -- be certain that cascade operator solely utilized to durations that settle for it
if how == "cascade" and any(
unit not in POLARS_DURATIONS_TO_IMMEDIATE_CHILD_MAPPING
for _, unit in decomposed_duration
):
increase ValueError(
(
"You requested a cascade situation on an invalid "
"length. Durations supporting cascade: "
f"{checklist(POLARS_DURATIONS_TO_IMMEDIATE_CHILD_MAPPING.keys())}"
)
)
rule_metadata = {
"operator": operator_method,
"decomposed_duration": decomposed_duration,
"how": how,
}
We now have, for every sample, dictionaries for find out how to outline the principles for every of it’s elements. So if we went for an advanced instance:
sample = '==1m>6d6h' # take away if month = Jan, and day > 6 and hour > 6
# parsed sample
[
[
{
"operator": "eq",
"decomposed_duration": [(1, "m")],
"how": "easy"
},
{
"operator": "gt",
"decomposed_duration": [(6, "d"), (6, "h")],
"how": "easy"
}
]
]
Discover {that a} single sample could be break up into a number of metadata dicts as a result of it may be composed of a number of durations and operations.
Creating Guidelines from metadata
Having created metadata for every sample, now comes the enjoyable a part of creating Polars guidelines!
Do not forget that inside every sample, we apply an AND situation, however throughout patterns we apply an OR situation. So within the easiest case, we want a wrapper that may take an inventory of all of the metadata for a selected sample, then apply the and situation to it. We are able to retailer this expression in an inventory alongside the expressions for all the opposite patterns, earlier than making use of the OR situation.
# -- dictionary to comprise every unit together with the polars technique to extract it's worth
UNIT_TO_POLARS_METHOD_MAPPING = {
"d": "day",
"h": "hour",
"m": "minute",
"s": "second",
"ms": "millisecond",
"us": "microsecond",
"ns": "nanosecond",
"wd": "weekday",
}
patterns = ["==6d<6h6s"]
patterns_metadata = get_rule_metadata_from_patterns(patterns)
# -- create an expression for the rule sample
pattern_metadata = patterns_metadata[0] # checklist of size two
# -- let's contemplate the situation for ==6d
situation = pattern_metadata[0]
decomposed_duration = situation["decomposed_duration"] # [(6, 'd')]
operator = situation["operator"] # eq
situations = [
getattr( # apply the operator method, e.g. pl.col("date").dt.hour().eq(value)
getattr( # get the value of the unit, e.g. pl.col("date").dt.hour()
pl.col(time_column).dt,
UNIT_TO_POLARS_METHOD_MAPPING[unit],
)(),
operator,
)(worth) for worth, unit in decomposed_duration # for every unit individually
]
# -- lastly, we combination the separate situations utilizing an AND situation
final_expression = situations.pop()
for expression in situations:
final_expression = getattr(final_expression, 'and_')(expression)
This appears to be like complicated… however we will convert bits of it into capabilities and the ultimate code appears to be like fairly clear and readable:
guidelines = [] # checklist to retailer expressions for every time sample
for rule_metadata in patterns_metadata:
rule_expressions = []
for situation in rule_metadata:
how = situation["how"]
decomposed_duration = situation["decomposed_duration"]
operator = situation["operator"]
if how == "easy":
expression = generate_polars_condition( # operate to do the ultimate mixture of expressions
[
self._generate_simple_condition(
unit, value, operator
) # this is the complex "getattr" code
for value, unit in decomposed_duration
],
"and_",
)
rule_expressions.append(expression)
rule_expression = generate_polars_condition(
rule_expressions, "and_"
)
guidelines.append(rule_expression)
overall_rule_expression = generate_polars_condition(
guidelines, "or_"
).not_() # we should negate as a result of we're filtering!
Creating Guidelines for the cascade operator
Within the above code, I had an if situation just for the “easy” situations… how can we do the cascade situations?
Keep in mind from our dialogue above {that a} sample of “>6h*” means:
hour > 6 OR (hour == 6 AND any(min, s, ms, and so forth… > 0)
So what we want, is to know for every unit, what the following smaller items are.
E.g. if I had “>6d*”, I ought to know to incorporate “hour” in my any situation, thus:
day > 6 OR (day == 6 AND any(hr, min, s, ms, and so forth… > 0)
That is simply achieved utilizing a dictionary that maps every unit to its “subsequent” smaller unit. E.g.: day → hour, hour → second, and so forth…
POLARS_DURATIONS_TO_IMMEDIATE_CHILD_MAPPING = {
"y": {"subsequent": "mo", "begin": 1},
"mo": {"subsequent": "d", "begin": 1},
"d": {"subsequent": "h", "begin": 0},
"wd": {"subsequent": "h", "begin": 0},
"h": {"subsequent": "m", "begin": 0},
"m": {"subsequent": "s", "begin": 0},
"s": {"subsequent": "ms", "begin": 0},
"ms": {"subsequent": "us", "begin": 0},
"us": {"subsequent": "ns", "begin": 0},
}
The beginning worth is important as a result of the any situation isn’t all the time > 0. As a result of if I wish to filter any values > February, then 2023–02–02 ought to be part of it, however not 2023–02–01.
With this dictionary in thoughts, we will then simply create the any situation:
# -- sample instance: >6h* cascade
simple_condition = self._generate_simple_condition(
unit, worth, operator
) # generate the straightforward situation, e.g. hour>6
all_conditions = [simple_condition]
if operator == "gt": # cascade solely impacts > operator
equality_condition = self._generate_simple_condition(
unit, worth, "eq"
) # generate hour==6
child_unit_conditions = []
child_unit_metadata = (
POLARS_DURATIONS_TO_IMMEDIATE_CHILD_MAPPING.get(unit, None)
) # get the subsequent smallest unit, e.g. minute
whereas child_unit_metadata just isn't None:
start_value = child_unit_metadata["start"]
child_unit = child_unit_metadata["next"]
child_unit_condition = self._generate_simple_condition(
child_unit, start_value, "gt"
) # generate minute > 0
child_unit_conditions.append(child_unit_condition)
child_unit_metadata = (
POLARS_DURATIONS_TO_IMMEDIATE_CHILD_MAPPING.get(
child_unit, None
)
) # now go on to seconds, and so forth...
cascase_condition = generate_polars_condition(
[
equality_condition, # and condition for the hour unit
generate_polars_condition(child_unit_conditions, "or_"), # any condition for all the child units
],
"and_",
)
all_conditions.append(cascase_condition)
# -- ultimate situation is hour>6 AND the cascade situation
overall_condition = generate_polars_condition(all_conditions, "or_")
The Larger Image
A processor like this isn’t simply helpful for ad-hoc evaluation. It may be a core element your information processing pipelines. One actually helpful use case for me is to make use of it together with resampling. A straightforward filtering step would allow me to straightforward calculate metrics on time collection with common disruptions, or common downtimes.
Additional, with just a few easy modifications I can lengthen this processor to permit straightforward labelling of my time collection. This enables me so as to add regressors to bits that I do know behave otherwise, e.g. if I’m modelling a time collection that jumps at particular hours, I can add a step regressor to solely these components.
Concluding Remarks
On this article I outlined a processor that allows straightforward, versatile and concise time collection filtration on Polars datasets. The logic mentioned could be prolonged to your favorite information body processing library, resembling Pandas with some minor modifications.
Not solely is the processor helpful for ad-hoc time collection evaluation, however it may be the spine of knowledge processing if chained with different operations resembling resampling, or if used to create further options for modelling.
I’ll conclude with some extensions that I keep in mind to make the code even higher:
- I’m pondering of making a brief lower to outline “weekend”, e.g. “==we”. This fashion I don’t wouldn’t have to explicitly outline “>=6wd” which could be much less clear
- With correct design, I feel it’s attainable to allow the addition of customized time identifiers. For instance “==eve” to indicate night, the time for which could be person outlined.
- I’m positively going so as to add assist for merely labelling the information, versus filtering it
- And I’m going so as to add assist for having the ability to outline the boundaries as “preserve”, e.g. as an alternative of defining [“<6h”, “>=20hr”] I can do [“>=6h<20hr”]
The place to seek out the code
This venture is in its infancy, so objects could transfer round. As of 23.05.2024, you could find the FilterDataBasedOnTime beneath mix_n_match/principal.py .
GitHub – namiyousef/mix-n-match: repository for processing dataframes
All code, information and pictures by writer until specified in any other case
Intuitive Temporal DataFrame Filtration was initially revealed in In the direction of Knowledge Science on Medium, the place persons are persevering with the dialog by highlighting and responding to this story.
[ad_2]