Learn and write dataframes as much as ten instances quicker than Parquet with StaticFrame NPZ

The Apache Parquet format supplies an environment friendly binary illustration of columnar desk information, as seen with widespread use in Apache Hadoop and Spark, AWS Athena and Glue, and Pandas DataFrame serialization. Whereas Parquet affords broad interoperability with efficiency superior to textual content codecs (comparable to CSV or JSON), it’s as a lot as ten instances slower than NPZ, an alternate DataFrame serialization format launched in StaticFrame.

StaticFrame (an open-source DataFrame library of which I’m an writer) builds upon NumPy NPY and NPZ codecs to encode DataFrames. The NPY format (a binary encoding of array information) and the NPZ format (zipped bundles of NPY recordsdata) are outlined in a NumPy Enhancement Proposal from 2007. By extending the NPZ format with specialised JSON metadata, StaticFrame supplies an entire DataFrame serialization format that helps all NumPy dtypes.

This text extends work first introduced at PyCon USA 2022 with additional efficiency optimizations and broader benchmarking.

DataFrames will not be simply collections of columnar information with string column labels, comparable to present in relational databases. Along with columnar information, DataFrames have labelled rows and columns, and people row and column labels will be of any sort or (with hierarchical labels) many sorts. Additional, it is not uncommon to retailer metadata with a title attribute, both on the DataFrame or on the axis labels.

As Parquet was initially designed simply to retailer collections of columnar information, the total vary of DataFrame traits just isn’t straight supported. Pandas provides this extra data by including JSON metadata into the Parquet file.

Additional, Parquet helps a minimal collection of sorts; the total vary of NumPy dtypes just isn’t straight supported. For instance, Parquet doesn’t natively help unsigned integers or any date sorts.

Whereas Python pickles are able to effectively serializing DataFrames and NumPy arrays, they’re solely appropriate for short-term caches from trusted sources. Whereas pickles are quick, they will change into invalid on account of code adjustments and are insecure to load from untrusted sources.

One other various to Parquet, originating within the Arrow undertaking, is Feather. Whereas Feather helps all Arrow sorts and succeeds in being quicker than Parquet, it’s nonetheless no less than two instances slower studying DataFrames than NPZ.

Parquet and Feather help compression to scale back file dimension. Parquet defaults to utilizing “snappy” compression, whereas Feather defaults to “lz4”. Because the NPZ format prioritizes efficiency, it doesn’t but help compression. As shall be proven beneath, NPZ outperforms each compressed and uncompressed Parquet recordsdata by important components.

Quite a few publications provide DataFrame benchmarks by testing only one or two datasets. McKinney and Richardson (2020) is an instance, the place two datasets, Fannie Mae Mortgage Efficiency and NYC Yellow Taxi Journey information, are used to generalize about efficiency. Such idiosyncratic datasets are inadequate, as each the form of the DataFrame and the diploma of columnar sort heterogeneity can considerably differentiate efficiency.

To keep away from this deficiency, I evaluate efficiency with a panel of 9 artificial datasets. These datasets fluctuate alongside two dimensions: form (tall, sq., and extensive) and columnar heterogeneity (columnar, combined, and uniform). Form variations alter the distribution of components between tall (e.g., 10,000 rows and 100 columns), sq. (e.g., 1,000 rows and columns), and extensive (e.g., 100 rows and 10,000 columns) geometries. Columnar heterogeneity variations alter the variety of sorts between columnar (no adjoining columns have the identical sort), combined (some adjoining columns have the identical sort), and uniform (all columns have the identical sort).

The frame-fixtures library defines a domain-specific language to create deterministic, randomly-generated DataFrames for testing; the 9 datasets are generated with this software.

To show a number of the StaticFrame and Pandas interfaces evaluated, the next IPython session performs primary efficiency checks utilizing %time. As proven beneath, a sq., uniformly-typed DataFrame will be written and skim with NPZ many instances quicker than uncompressed Parquet.

>>> import numpy as np
>>> import static_frame as sf
>>> import pandas as pd
>>> # an sq., uniform float array
>>> array = np.random.random_sample((10_000, 10_000))
>>> # write peformance
>>> f1 = sf.Body(array)
>>> %time f1.to_npz('/tmp/body.npz')
CPU instances: consumer 710 ms, sys: 396 ms, complete: 1.11 s
Wall time: 1.11 s
>>> df1 = pd.DataFrame(array)
>>> %time df1.to_parquet('/tmp/df.parquet', compression=None)
CPU instances: consumer 6.82 s, sys: 900 ms, complete: 7.72 s
Wall time: 7.74 s
>>> # learn efficiency
>>> %time f2 = f1.from_npz('/tmp/body.npz')
CPU instances: consumer 2.77 ms, sys: 163 ms, complete: 166 ms
Wall time: 165 ms
>>> %time df2 = pd.read_parquet('/tmp/df.parquet')
CPU instances: consumer 2.55 s, sys: 1.2 s, complete: 3.75 s
Wall time: 866 ms

Efficiency checks offered beneath lengthen this primary method by utilizing frame-fixtures for systematic variation of form and sort heterogeneity, and common outcomes over ten iterations. Whereas {hardware} configuration will have an effect on efficiency, relative traits are retained throughout various machines and working techniques. For all interfaces the default parameters are used, apart from disabling compression as wanted. The code used to carry out these checks is accessible at GitHub.

Learn Efficiency

As information is mostly learn extra usually then it’s written, learn efficiency is a precedence. As proven for all 9 DataFrames of 1 million (1e+06) components, NPZ considerably outperforms Parquet and Feather with each fixture. NPZ learn efficiency is over ten instances quicker than compressed Parquet. For instance, with the Uniform Tall fixture, compressed Parquet studying is 21 ms in comparison with 1.5 ms with NPZ.

The chart beneath reveals processing time, the place decrease bars correspond to quicker efficiency.

This spectacular NPZ efficiency is retained with scale. Transferring to 100 million (1e+08) components, NPZ continues to carry out no less than twice as quick as Parquet and Feather, no matter if compression is used.

Write Efficiency

In writing DataFrames to disk, NPZ outperforms Parquet (each compressed and uncompressed) in all eventualities. For instance, with the Uniform Sq. fixture, compressed Parquet writing is 200 ms in comparison with 18.3 ms with NPZ. NPZ write efficiency is mostly similar to uncompressed Feather: in some eventualities NPZ is quicker, in others, Feather is quicker.

As with learn efficiency, NPZ write efficiency is retained with scale. Transferring to 100 million (1e+08) components, NPZ continues to be no less than twice as quick as Parquet, no matter if compression is used or not.

Idiosyncratic Efficiency

As an extra reference, we may also benchmark the identical NYC Yellow Taxi Journey information (from January 2010) utilized in McKinney and Richardson (2020). This dataset incorporates nearly 300 million (3e+08) components in a tall, heterogeneously typed DataFrame of 14,863,778 rows and 19 columns.

NPZ learn efficiency is proven to be round 4 instances quicker than Parquet and Feather (with or with out compression). Whereas NPZ write efficiency is quicker than Parquet, Feather writing right here is quickest.

File Measurement

As proven beneath for a million (1e+06) factor and 100 million (1e+08) factor DataFrames, uncompressed NPZ is mostly equal in dimension on disk to uncompressed Feather and all the time smaller than uncompressed Parquet (typically smaller than compressed Parquet too). As compression supplies solely modest file-size reductions for Parquet and Feather, the good thing about uncompressed NPZ in velocity would possibly simply outweigh the price of better dimension.

StaticFrame shops information as a set of 1D and 2D NumPy arrays. Arrays symbolize columnar values, in addition to variable-depth index and column labels. Along with NumPy arrays, details about element sorts (i.e., the Python class used for the index and columns), in addition to the element title attributes, are wanted to totally reconstruct a Body. Fully serializing a DataFrame requires writing and studying these elements to a file.

DataFrame elements will be represented by the next diagram, which isolates arrays, array sorts, element sorts, and element names. This diagram shall be used to show how an NPZ encodes a DataFrame.

The elements of that diagram map to elements of a Body string illustration in Python. For instance, given a Body of integers and Booleans with hierarchical labels on each the index and columns (downloadable through GitHub with StaticFrame’s WWW interface), StaticFrame supplies the next string illustration:

>>> body = sf.Body.from_npz(sf.WWW.from_file('https://github.com/static-frame/static-frame/uncooked/grasp/doc/supply/articles/serialize/body.npz', encoding=None))
>>> body
<Body: p>
<IndexHierarchy: q>       information    information    information    legitimate  <<U5>
A       B       C       *      <<U1>
<IndexHierarchy: r>
2012-03             x     5       4       7       False
2012-03             y     9       1       8       True
2012-04             x     3       6       2       True
<datetime64[M]>     <<U1> <int64> <int64> <int64> <bool>

The elements of the string illustration will be mapped to the DataFrame diagram by shade:

Encoding an Array in NPY

A NPY shops a NumPy array as a binary file with six elements: (1) a “magic” prefix, (2) a model quantity, (3) a header size and (4) header (the place the header is a string illustration of a Python dictionary), and (5) padding adopted by (6) uncooked array byte information. These elements are proven beneath for a three-element binary array saved in a file named “__blocks_1__.npy”.

Given a NPZ file named “body.npz”, we will extract the binary information by studying the NPY file from the NPZ with the usual library’s ZipFile:

>>> from zipfile import ZipFile
>>> with ZipFile('/tmp/body.npz') as zf: print(zf.open('__blocks_1__.npy').learn())
b'x93NUMPYx01x006x00b1","fortran_order":True,"form":(3,)    nx00x01x01

As NPY is effectively supported in NumPy, the np.load() perform can be utilized to transform this file to a NumPy array. Because of this underlying array information in a StaticFrame NPZ is well extractable by various readers.

>>> with ZipFile('/tmp/body.npz') as zf: print(repr(np.load(zf.open('__blocks_1__.npy'))))
array([False,  True,  True])

As a NPY file can encode any array, giant two-dimensional arrays will be loaded from contiguous byte information, offering wonderful efficiency in StaticFrame when a number of contiguous columns are represented by a single array.

Constructing a NPZ File

A StaticFrame NPZ is a typical uncompressed ZIP file that incorporates array information in NPY recordsdata and metadata (containing element sorts and names) in a JSON file.

Given the NPZ file for the Body above, we will checklist its contents with ZipFile. The archive incorporates six NPY recordsdata and one JSON file.

>>> with ZipFile('/tmp/body.npz') as zf: print(zf.namelist())
['__values_index_0__.npy', '__values_index_1__.npy', '__values_columns_0__.npy', '__values_columns_1__.npy', '__blocks_0__.npy', '__blocks_1__.npy', '__meta__.json']

The illustration beneath maps these recordsdata to elements of the DataFrame diagram.

StaticFrame extends the NPZ format to incorporate metadata in a JSON file. This file defines title attributes, element sorts, and depth counts.

>>> with ZipFile('/tmp/body.npz') as zf: print(zf.open('__meta__.json').learn())
b'{"__names__": ["p", "r", "q"], "__types__": ["IndexHierarchy", "IndexHierarchy"], "__types_index__": ["IndexYearMonth", "Index"], "__types_columns__": ["Index", "Index"], "__depths__": [2, 2, 2]}'

Within the illustration beneath, elements of the __meta__.json file are mapped to elements of the DataFrame diagram.

As a easy ZIP file, instruments to extract the contents of a StaticFrame NPZ are ubiquitous. Then again, the ZIP format, given its historical past and broad options, incurs efficiency overhead. StaticFrame implements a customized ZIP reader optimized for NPZ utilization, which contributes to the superb learn efficiency of NPZ.

The efficiency of DataFrame serialization is crucial to many functions. Whereas Parquet has widespread help, its generality compromises sort specificity and efficiency. StaticFrame NPZ can learn and write DataFrames as much as ten-times quicker than Parquet with or with out compression, with comparable (or solely modestly bigger) file sizes. Whereas Feather is a horny various, NPZ learn efficiency continues to be usually twice as quick as Feather. If information I/O is a bottleneck (and it usually is), StaticFrame NPZ affords an answer.