Essentially the most fascinating issues concerning the new launch

pandas 2.2 was launched on January twenty second 2024. Let’s check out the issues this launch introduces and the way it will assist us to enhance our pandas workloads. It features a bunch of enhancements that can enhance the consumer expertise.

pandas 2.2 introduced just a few extra enhancements that depend on the Apache Arrow ecosystem. Moreover, we added deprecations for modifications which might be essential to make Copy-on-Write the default in pandas 3.0. Let’s dig into what this implies for you. We are going to take a look at an important modifications intimately.

I’m a part of the pandas core workforce. I’m an open supply engineer for Coiled the place I work on Dask, together with bettering the pandas integration.

Improved PyArrow help

Now we have launched PyArrow backed DataFrame in pandas 2.0 and continued to enhance the mixing since then to allow a seamless integration into the pandas API. pandas has accessors for sure dtypes that allow specialised operations, just like the string accessor, that gives many string strategies. Traditionally, record and structs have been represented as NumPy object dtype, which made working with them fairly cumbersome. The Arrow dtype backend now permits tailor-made accessors for lists and structs, which makes working with these objects quite a bit simpler.

Let’s take a look at an instance:

import pyarrow as pa
sequence = pd.Sequence(
[
{"project": "pandas", "version": "2.2.0"},
{"project": "numpy", "version": "1.25.2"},
{"project": "pyarrow", "version": "13.0.0"},
],
dtype=pd.ArrowDtype(
pa.struct([
("project", pa.string()),
("version", pa.string()),
])
),
)

It is a sequence that accommodates a dictionary in each row. Beforehand, this was solely attainable with NumPy object dtype and accessing parts from these rows required iterating over them. The struct accessor now permits direct entry to sure attributes:

sequence.struct.subject("mission")
0     pandas
1      numpy
2    pyarrow
Title: mission, dtype: string[pyarrow]

The following launch will convey a CategoricalAccessor primarily based on Arrow varieties.

Integrating the Apache ADBC Driver

Traditionally, pandas relied on SqlAlchemy to learn knowledge from an Sql database. This labored very reliably, however it was very sluggish. Alchemy reads the information row-wise, whereas pandas has a columnar structure, which makes studying and transferring the information right into a DataFrame slower than essential.

The ADBC Driver from the Apache Arrow mission permits customers to learn knowledge in a columnar structure, which brings large efficiency enhancements. It reads the information and shops them into an Arrow desk, which is used to transform to a pandas DataFrame. You can also make this conversion zero-copy, if you happen to set dtype_backend="pyarrow" for read_sql.

Let’s take a look at an instance:

import adbc_driver_postgresql.dbapi as pg_dbapi
df = pd.DataFrame(
[
[1, 2, 3],
[4, 5, 6],
],
columns=['a', 'b', 'c']
)
uri = "postgresql://postgres:postgres@localhost/postgres"
with pg_dbapi.join(uri) as conn:
df.to_sql("pandas_table", conn, index=False)
# for round-tripping
with pg_dbapi.join(uri) as conn:
df2 = pd.read_sql("pandas_table", conn)

The ADBC Driver at present helps Postgres and Sqlite. I’d suggest everybody to modify over to this driver if you happen to use Postgres, the driving force is considerably quicker and fully avoids round-tripping by means of Python objects, thus preserving the database varieties extra reliably. That is the function that I’m personally most enthusiastic about.

Including case_when to the pandas API

Coming from Sql to pandas, customers usually miss the case-when syntax that gives a simple and clear technique to create new columns conditionally. pandas 2.2 provides a brand new case_when technique, that’s outlined on a Sequence. It operates equally to what Sql does.

Let’s take a look at an instance:

df = pd.DataFrame(dict(a=[1, 2, 3], b=[4, 5, 6]))
default=pd.Sequence('default', index=df.index)
default.case_when(
caselist=[
(df.a == 1, 'first'),
(df.a.gt(1) & df.b.eq(5), 'second'),
],
)

The tactic takes an inventory of circumstances which might be evaluated sequentially. The brand new object is then created with these values in rows the place the situation evaluates to True. The tactic ought to make it considerably simpler for us to create conditional columns.

Copy-on-Write

Copy-on-Write was initially launched in pandas 1.5.0. The mode will grow to be the default conduct with 3.0, which is hopefully the subsequent pandas launch. Because of this we have now to get our code right into a state the place it’s compliant with the Copy-on-Write guidelines. pandas 2.2 launched deprecation warnings for operations that can change conduct.

df = pd.DataFrame({"x": [1, 2, 3]})
df["x"][df["x"] > 1] = 100

It will now elevate a FutureWarning.

FutureWarning: ChainedAssignmentError: behaviour will change in pandas 3.0!
You're setting values by means of chained task. Presently this works in sure circumstances, however when 
utilizing Copy-on-Write (which is able to grow to be the default behaviour in pandas 3.0) this can by no means work to 
replace the unique DataFrame or Sequence, as a result of the intermediate object on which we're setting 
values will behave as a replica. A typical instance is when you're setting values in a column of a 
DataFrame, like:
df["col"][row_indexer] = worth
Use `df.loc[row_indexer, "col"] = values` as a substitute, to carry out the task in a single step and 
guarantee this retains updating the unique `df`.

I wrote an earlier submit that goes into extra element about how one can migrate your code and what to anticipate. There’s a further warning mode for Copy-on-Write that can elevate warnings for all circumstances that change conduct:

pd.choices.mode.copy_on_write = "warn"

Most of these warnings are solely noise for almost all of pandas customers, which is the explanation why they’re hidden behind an possibility.

pd.choices.mode.copy_on_write = "warn"
df = pd.DataFrame({"a": [1, 2, 3]})
view = df["a"]
view.iloc[0] = 100

It will elevate a prolonged warning explaining what is going to change:

FutureWarning: Setting a worth on a view: behaviour will change in pandas 3.0.
You're mutating a Sequence or DataFrame object, and at present this mutation will
even have impact on different Sequence or DataFrame objects that share knowledge with this
object. In pandas 3.0 (with Copy-on-Write), updating one Sequence or DataFrame object
won't ever modify one other.

The quick abstract of that is: Updating view won’t ever replace df, it doesn’t matter what operation is used. That is almost definitely not related for many.

I’d suggest enabling the mode and checking the warnings briefly, however to not pay an excessive amount of consideration to them if you’re snug that you’re not counting on updating two totally different objects without delay.

I’d suggest testing the migration information for Copy-on-Write that explains the mandatory modifications in additional element.

Upgrading to the brand new model

You may set up the brand new pandas model with:

pip set up -U pandas

Or:

mamba set up -c conda-forge pandas=2.2

This will provide you with the brand new launch in your atmosphere.

Conclusion

We’ve checked out a few enhancements that can enhance efficiency and consumer expertise for sure features of pandas. Essentially the most thrilling new options will are available pandas 3.0, the place Copy-on-Write will likely be enabled by default.

Thanks for studying. Be at liberty to achieve out to share your ideas and suggestions.