TL;DR
Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars has released version 2.0, making its streaming engine the default for collecting lazy queries and enabling initial spill-to-disk support. The project also expands SQL support and reports strong TPC-H and TPC-DS benchmark results, while noting test limitations and a scaling issue on large machines.
Polars has shipped version 2.0, making its streaming engine the default when users collect a lazy query and enabling an initial form of spill-to-disk processing. The release also expands SQL support and adds a Map data type; these changes affect memory use, row-order behavior and how some workloads can run, so users may need to review existing query assumptions.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says this can bring memory and performance improvements on many queries. However, streaming does not preserve observable row order by default for some operations, including joins, group-bys and unpivots. Users who need order preserved can set maintain_order=True for supported operations.
Version 2.0 also enables out-of-core processing, which allows supported operations to spill data to disk when memory use grows. The release post says spilling begins at about 80% of RAM and the default disk budget is 64 GB; both settings may need adjustment for particular systems. Current support includes sorting, window functions and many expressions. Joins and group-bys are not yet listed as supported for spill-to-disk.
Polars describes SQL as a first-class interface in this release and cites optimizer and engine work including join reordering, common-subplan elimination and dynamic predicates or bloom filters. The release adds a native Map dtype for key-value data, with operations such as key lookup, checking for a key, and retrieving keys or values. It also emphasizes stricter handling of data types and explicitness, which the project says can provide faster feedback.
Streaming Changes Query Defaults
The most consequential change for existing users is not a new syntax feature but a new execution default. Streaming can reduce the memory pressure of larger queries, while spilling supported operations to disk may let some workloads finish when they exceed available RAM. That could make Polars more practical on machines where a query previously required a larger memory allocation, though the release does not claim that every operation can now run out of core.
The trade-off is that row order is not guaranteed by default in certain operations. Applications that depend on the previous observable ordering should check their results and use the documented order-preservation option where needed. The new SQL positioning may also make Polars relevant to teams that prefer SQL workflows, but benchmark outcomes alone do not establish which engine will be fastest for a given workload.
Polars 2.0 data processing software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Polars Tested SQL Performance
To support its performance claims, the Polars team compared its SQL engine with DuckDB 1.5.6, DuckDB 2.0 alpha and DataFusion 54.0.0 using queries and data derived from TPC-H and TPC-DS. Tests ran on two AWS machine types: one with 16 vCPUs and 32 GB of RAM, and another with 192 vCPUs and 384 GB. The project reports that Polars and both DuckDB versions completed all queries; DataFusion timed out on one TPC-DS query, timed out once on another, and ran out of memory on a TPC-H query on the smaller machine. The affected queries were excluded from the reported results for all engines.
Polars says its default configuration was fastest on all but one of the benchmark groupings it tested, and that a 32-core-limited configuration was competitive or faster across them. These are vendor-reported benchmark findings, not an independent evaluation. The team published a repository intended to help others reproduce the tests and disclosed that Polars has overhead when scaling to 192 threads, which hurt results on small-data queries. It said it had diagnosed the issue and hoped to address it in a later release.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars release post
As an affiliate, we earn on qualifying purchases.
Limits of the New Execution Model
The release material does not specify a calendar date for the announcement. It also does not provide enough information here to establish how the reported results translate to other hardware, datasets or query mixes. The benchmark post gives test conditions and a reproduction repository, but the results remain Polars’ own measurements.
Out-of-core support is described as an initial release: joins and group-bys are still on the roadmap, and the team says the approximate 80% RAM spill threshold may need tuning. The post also flags a large-machine scaling issue and says a fix is hoped for in a future release, without identifying a release number or schedule. Users should verify the behavior of their own queries, especially where row order or memory limits matter.
As an affiliate, we earn on qualifying purchases.
Further Spill Support and Tuning
Polars says it plans to extend out-of-core processing to joins and group-bys. The team also hopes to fix the overhead it observed when scaling to 192 threads in a later release, but it has not given a timetable in the source material. Meanwhile, users can consult the published benchmark repository, test version 2.0 against their own workloads, and review any queries that rely on row order before adopting the new defaults.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
Collecting a LazyFrame now defaults to the streaming engine. The release also enables initial spill-to-disk support and expands SQL features.
Does streaming preserve row order?
Not by default for some operations, including joins, group-bys and unpivots. Polars says users who require observable order can set maintain_order=True where supported.
Which operations can spill data to disk?
The release lists sorting, window functions and many expressions as supported. Joins and group-bys are not yet included; Polars says those are planned for future support.
Are the performance claims independently verified?
The cited TPC-H and TPC-DS results come from Polars’ own tests. The team published a repository to support replication, but the release post does not present an independent benchmark.
Source: hn
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
