Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

dataengineeringcentral.substack.com

Post reply on HN
Print view View original

81–90 of 112 posts

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#81

Post by rmnclmnt »

esafak wrote: post #9
mynameisash wrote: post #8

Earlier quoted context omitted.

> It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!" and "you can throw more nodes at it easy-peasy!" And yet, there are so many cases where you can just do things with better tools. Like the time a junior engineer asked for help processing 1…

I used pySpark some time ago when it was introduced to my company at the time and I realized that it was slow when you used python libraries in the UDFs rather than pySpark's own functions.

Yes using Python UDFs within Spark pipelines are a hog! That’s because the entire Python context is serialized with cloudpickle and sent over the wire to the executor nodes! (It can represent a few GB of serialized data depending on the UDF and driver process Python context)

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#82

Post by rmnclmnt »

pu_pe wrote: post #45

In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.

Until the product manager ask for the bill… then all of a sudden things get reconsidered

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#83

Post by faizshah »

I had to do something like this for a few TB of json recently. The unique thing about this workload was it was a ton of small 10-20mb files.

I found that clickhouse was the fastest, but duckdb was the simplest to work with it usually just works. DuckDB was close enough to the max performance from clickhouse.

I tried flink & pyspark but they were way slower (like 3-5x) than clickhouse and the code was kind of annoying. Dask and Ray were also way too slow, but dask’s parallelism was easy to code but it was just too slow. I also tried Datafusion and polars but clickhouse ended up being faster.

These days I would recommend starting with DuckDB or Clickhouse for most workloads just cause it’s the easiest to work with AND has good performance. Personally I switched to using DuckDB instead of polars for most things where pandas is too slow.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#84

Post by faizshah »

throwaway-aws9 wrote: post #35

650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-...

This isn’t true anymore we are way beyond 2014 Hadoop (what the blog post is about) at this point.

Go try doing an aggregation of 650gb of json data using normal CLI tools vs duckdb or clickhouse. These tools are pipelining and parallelizing in a way that isn’t easy to do with just GNU Parallel (trust me, I’ve tried).

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#85

Post by Scubabear68 »

luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

This is a really good observation, and matches something I had to learn painfully over 30 years ago. At a Wall Street bank, we were trying to really push the limits with some middleware, and my mentor at the time very quietly suggested "before you test your system's performance, understand the theoretical maximum of your setup first with no work".

The gist was - find your resource limits and saturate them and see what the best possible performance could be, then measure your system, and you can express it as a percentage of optimal. Or if you can't directly test/saturate your limits at least be aware of them.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#86

Post by thinkharderdev »

adammarples wrote: post #80
Demiurge wrote: post #38

Earlier quoted context omitted.

What if it was 650TB? This article is obviously a microbenchmark. I work with much larger datasets, and neither awk nor DBD would make a difference to the overall architecture. You need a data catalog, and you need a clusters of jobs at scale, regardless of a data format library, or libraries.

I think the entire point of the article (reading forward a bit through the linked redshift files posts) is that almost nobody in the world uses datasets bigger than 100Tb, that when they do, they use a small subset anyway, and that 650Gb is a pretty reasonable approximation of the entire dataset most companies are even working with. Certainly in my experience as a data engineer, they're not often in the many terabyte…

> It's good to know that OOTB duckdb can replace snowflake et all in these situations, especially with how expensive they are.

Does this article demonstrate that though? I get, and agree, that a lot of people are using "big data" tools for datasets that are way too small to require it. But this article consists of exactly one very simple aggregation query. And even then it takes 16m to run (in the best case). As others have mentioned the long execution time is almost certainly dominated by IO because of limited network bandwidth, but network bandwidth is one of the resources you get more of in a distributed computing environment.

But my bigger issue is just that real analytical queries are often quite a bit more complicated than a simple count by timestamp. As soon as you start adding non-trivial compute to query, or multiple joins (and g*d forbid you have a nested-loop join in there somewhere), or sorting then the single node execution time is going to explode.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#88

Post by patwolf »

pu_pe wrote: post #45

In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.

I worked at a company that dumped Databricks once the first bill came. I guess it was an order of magnitude more expensive than what they expected. It was less expensive to rebuild the pipeline from scratch with a different product.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#89

Post by nijave »

kccqzy wrote: post #14
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

10Gbps only? At Google where this type of processing would automatically be distributed, machines had 400Gbps NICs, not to mention other innovations like better TCP congestion control algorithms. No wonder people are tired of distributed computing.

They do at AWS, too but op paid for a small VM

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#90

Post by nijave »

mynameisash wrote: post #8
esafak wrote: post #2

If I understand correctly, polars relies on delta-rs for Delta Lake support, and that is what does not support Deletion vectors: https://github.com/delta-io/delta-rs/issues/1094 It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark.

> It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!" and "you can throw more nodes at it easy-peasy!" And yet, there are so many cases where you can just do things with better tools. Like the time a junior engineer asked for help processing 1…

Python isn't too bad if you swap in orjson instead of stdlib which is quite a bit slower

Wrangling multiprocess is still annoying tho

Post reply on HN