According to the post, the author’s talk at Latency Conference argues that the Python DataFrame library Pandas should go extinct—not the animal, but the software. The post explains that many analysts start with Excel, move to Pandas when data reaches the gigabyte range, and then encounter memory limits, slow computation, or frustration with Pandas’ baroque API once data grows into the tens of gigabytes. At that point, the typical response is to adopt distributed systems such as Spark, DataBricks, Snowflake, or Dask, which are marketed as solutions for “Big Data.” However, the author contends there is a sizable gap between the point where Pandas falters and the scale where distributed systems are genuinely required. This gap lies around the 100 GB mark and can be filled by modern, high‑performance, single‑machine tools, chiefly Polars and DuckDB. To support the claim that most workloads never reach the “Big Data” threshold, the post cites an Amazon Redshift fleet analysis published in 2024. Assuming an average row size of 1 KB and clusters of ten machines each ingesting data at 8 GB/s from S3, the analysis shows that 94.68 % of tables contain fewer than 100 GB of data and that 86.9 % of queries operate on 80 GB or less. Under these assumptions, the post concludes that the majority of users face “Medium Data” problems rather than true Big Data challenges. The post then introduces the alternatives. Polars is described as a Rust‑based DataFrame library that feels familiar to Pandas users but differs in its execution model: it uses lazy evaluation via scan_csv, processes data in chunks, and collects results with new_streaming=True, allowing an optimized query graph similar to a database. DuckDB is characterized as an in‑memory analytics database, essentially SQLite for analytics, that can handle medium‑sized workloads without the overhead of a cluster. To illustrate the performance differences, the post walks through the 1 Billion Row Challenge, where a CSV of weather station data must be processed to compute min, mean, and max per station. The Pandas implementation reads the entire file into memory, groups, and then aggregates. In contrast, the Polars version lazily scans the CSV, groups, and aggregates while streaming the data, demonstrating a different runtime approach. The fastest accepted implementation for that challenge ran in 1.5 seconds on a 32‑core, 128 GB RAM server, showing that single‑machine tools can meet demanding performance targets. The post ultimately suggests that for most analytical tasks, adopting Polars or DuckDB avoids the premature complexity and cost of distributed systems while still delivering acceptable speed and scalability.
Key facts
- 94.68% of Redshift tables contain <100 GB of data
- 86.9% of Redshift queries operate on ≤80 GB of data
- Polars uses lazy scan_csv and chunk-wise processing
- DuckDB is an in‑memory analytics DB akin to SQLite for analytics
- The 1 Billion Row Challenge fastest implementation ran in 1.5 seconds