Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

dataengineeringcentral.substack.com

Post reply on HN
Print view View original

11–20 of 112 posts

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#11

Post by tempest_ »

amluto wrote: post #10
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

It would be amusing to run this on a regular desktop computer or even a moderately nice laptop (with a fan - give it a chance!) and see how it does. 650GB will stream in quite quickly from any decent NVMe device, and those 8-16 cores might well be considerably faster than whatever cores the cloud machines are giving you. S3 is an amazingly engineered product, operates at truly impressive scale, is quite reasonably pr…

Everyone wants a data lake when what they have a is a data pond.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#12

Post by bushbaba »

luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

I'm kind of suprised they didn't choose an ec2 instance with higher throughput. S3 can totally eek out 100s of Gibps with the right setup.

BUT the author did say this is the simple stupid naive take, in which case DuckDB and Polars really shined.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#13

Post by switchbak »

amluto wrote: post #10
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

It would be amusing to run this on a regular desktop computer or even a moderately nice laptop (with a fan - give it a chance!) and see how it does. 650GB will stream in quite quickly from any decent NVMe device, and those 8-16 cores might well be considerably faster than whatever cores the cloud machines are giving you. S3 is an amazingly engineered product, operates at truly impressive scale, is quite reasonably pr…

Absolutely. I recently reworked a bunch of tests and found my desktop to outcompete our (larger, custom) Github Action runner by roughly 5x. And I expect this delta to increase a lot as you lean on the local I/O harder.

It really is shocking how much you're paying given how little you get. I certainly don't want to run a data center and handle all the scaling and complexity of such an endeavour. But wow, the tax you pay to have someone manage all that is staggering.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#14

Post by kccqzy »

luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

10Gbps only? At Google where this type of processing would automatically be distributed, machines had 400Gbps NICs, not to mention other innovations like better TCP congestion control algorithms. No wonder people are tired of distributed computing.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#15

Post by basilgohar »

kccqzy wrote: post #14
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

10Gbps only? At Google where this type of processing would automatically be distributed, machines had 400Gbps NICs, not to mention other innovations like better TCP congestion control algorithms. No wonder people are tired of distributed computing.

"At Google" is doing all the heavy lifting in your comment here, with all due respect. There is but one Google but remain millions of us who are not "At Google".

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#18

Post by garganzol »

jdnier wrote: post #7

DuckDb has a new "DuckLake" catalog format that would be another candidate to test. https://ducklake.select/

DuckLake format has an unresolved built-in chicken and egg conflict: it requires SQL database to represent its catalog. But this is what some people are running away from when they choose Parquet format in the first place. Parquet = easy, SQL = hard, adding SQL to Parquet makes the resulting format hard. I would expect a catalog to be in Parquet format as well, then it becomes something self-bootstrapping and usable.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#20

Post by 0cf8612b2e1e »

hnidiots3 wrote: post #19

650GB? We have 72PB IN S3, know people who have multiple EB in S3.

This is not a game of, “mine is bigger than yours”. Many many workloads in the wild are smaller than this.

Motherduck have a few posts about how few people have “big data”. https://motherduck.com/blog/redshift-files-hunt-for-big-data...

Post reply on HN