Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

dataengineeringcentral.substack.com

Post reply on HN
Print view View original

31–40 of 112 posts

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#31

Post by mrlongroots »

luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

Yep I think the value of the experiment is not clear.

You want to use Spark for a large dataset with multiple stages. In this case, their I/O bandwidth is 1GB/s from S3. CPU memory bandwidth is 100-200GB/s for a multi-stage job. Spark is a way to pool memory for a large dataset with multiple stages, and use cluster-internal network bandwidth to do shuffling instead of storage.

Maybe when you have S3 as your backend, the storage bandwidth bottleneck doesn't show up in perf, but it sure does show up in the bill. A crude rule of thumb: network bandwidth is 20X storage, main memory bandwidth is 20X network bandwidth, accelerator/GPU memory is 10X CPU. It's great that single-node DuckDB/Polars are that good, but this is like racing a taxiing aircraft against motorbikes.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#32

Post by dsp_person »

matt123456789 wrote: post #28
garganzol wrote: post #18

Earlier quoted context omitted.

DuckLake format has an unresolved built-in chicken and egg conflict: it requires SQL database to represent its catalog. But this is what some people are running away from when they choose Parquet format in the first place. Parquet = easy, SQL = hard, adding SQL to Parquet makes the resulting format hard. I would expect a catalog to be in Parquet format as well, then it becomes something self-bootstrapping and usable.

It is not a chicken and egg problem, it is just a requirement to have an RDBMS available for systems like DuckLake and Hive to store their catalogs in. Metadata is relatively small and needs to provide ACID r/w => great RDBMS use case.

What about file-based catalogs with Iceberg? Found one that puts it in a single json file: https://github.com/boringdata/boring-catalog

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#33

Post by srilman »

Hey everyone, I'm a software engineer at Eventual, the team behind Daft! Huge thanks to the op for the benchmark, we're a huge fan of your blog posts and this gave us some really useful insights. For context, Daft is a high-performance data processing engine for AI workloads that works both on single-node and distributed setups.

We're actively looking into the results of the benchmark and hope to share some of our findings soon. From initial results, we found a lot of potential optimizations we could make to our deltalake reader to improve parallelism and our groupby operator to improve pipelining for count aggregations. We're hoping to roll our these improvements over the next couple of releases.

If you're interested to learn more about our findings, check out our GitHub (https://github.com/Eventual-Inc/Daft) or follow us on Twitter (https://x.com/daftengine) and LinkedIn (https://www.linkedin.com/showcase/daftengine) for updates. Also if Daft sounds interesting to you, give us a try via pip install daft!

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#34

Post by saxenaabhi »

dsp_person wrote: post #32
matt123456789 wrote: post #28

Earlier quoted context omitted.

It is not a chicken and egg problem, it is just a requirement to have an RDBMS available for systems like DuckLake and Hive to store their catalogs in. Metadata is relatively small and needs to provide ACID r/w => great RDBMS use case.

What about file-based catalogs with Iceberg? Found one that puts it in a single json file: https://github.com/boringdata/boring-catalog

Then concurrency suffers since you have to have locks when you update files.

That's also why ducklake performs better than others.

For many use cases this trade-off is worth it.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#36

Post by layoric »

amluto wrote: post #10
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

It would be amusing to run this on a regular desktop computer or even a moderately nice laptop (with a fan - give it a chance!) and see how it does. 650GB will stream in quite quickly from any decent NVMe device, and those 8-16 cores might well be considerably faster than whatever cores the cloud machines are giving you. S3 is an amazingly engineered product, operates at truly impressive scale, is quite reasonably pr…

Totally true. I have a trusty old (like 2016 era) X99 setup that I use for 1.2TB of time series data hosted in a timescaledb PostGIS database. I can fetch all the data I need quickly to crunch on another local machine, and max out my aging network gear to experiment with different model training scenarios. It cost me ~$500 to build the machine, and it stays off when I'm not using it.

Much easier obviously dealing with a dataset that doesn't change, but doing the same in the cloud would just be throwing money away.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#37

Post by m00x »

co0lster wrote: post #5

650GB relates to size of parquet files which are compressed in reality it’s way more. 32 GB of parquet cannot fit in 32GB of RAM

You don't need it to if you just need specific columns. This is the advantage of columnar storage.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#38

Post by Demiurge »

throwaway-aws9 wrote: post #35

650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-...

What if it was 650TB? This article is obviously a microbenchmark. I work with much larger datasets, and neither awk nor DBD would make a difference to the overall architecture. You need a data catalog, and you need a clusters of jobs at scale, regardless of a data format library, or libraries.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#39

Post by willvarfar »

I often crunch 'biggish data' on a single node using duckdb (because I love using the modern style of painless and efficient SQL engines).

I don't use delta or iceberg (because I haven't needed to; I'm describing what I do, not what you can do :)), but rather just iterate over the underlying parquet files using filename listing or wildcarding. I often run queries on BigQuery and suck down the results to a bunch of ~1GB local parquet files - way bigger than RAM - that I can then mine in duckdb using wildcarding. Works great!

I'm in a world where I get into the weeds of 'this kind of aggregation works much faster on Bigquery than duckdb, or vice versa, so I'll split my job into this part of sql running on Bigquery then feeding into this part running in duckdb'. It's the fun end of data engineering.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#40

Post by otterley »

kccqzy wrote: post #14
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

10Gbps only? At Google where this type of processing would automatically be distributed, machines had 400Gbps NICs, not to mention other innovations like better TCP congestion control algorithms. No wonder people are tired of distributed computing.

You can get a 600Gbps interface on an Amazon EC2 instance (c8gn.48xlarge), if you’re willing to pay for it.
Post reply on HN