Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

dataengineeringcentral.substack.com

Post reply on HN
Print view View original

51–60 of 112 posts

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#51

Post by CraigJPerry »

Demiurge wrote: post #38
throwaway-aws9 wrote: post #35

650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-...

What if it was 650TB? This article is obviously a microbenchmark. I work with much larger datasets, and neither awk nor DBD would make a difference to the overall architecture. You need a data catalog, and you need a clusters of jobs at scale, regardless of a data format library, or libraries.

At 650tb it's not a memory bound problem:

working memory requirements

    1. Assume date is 8 bytes
    2. Assume 64bit counters
So for each date in the dataset we need 16 bytes to accumulate the result.

That's ~180 years worth of daily post counts per gb ram - but the dataset in the post was just 1 year.

This problem should be mostly network limited in the OP's context, decompressing snappy compressed parquet should be circa 1gb/sec. The "work" of parsing a string to a date and accumulating isn't expensive compared to snappy decompression.

I don't have a handle on the 33% longer runtime difference between duckdb and polars here.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#52

Post by miohtama »

willvarfar wrote: post #39

I often crunch 'biggish data' on a single node using duckdb (because I love using the modern style of painless and efficient SQL engines). I don't use delta or iceberg (because I haven't needed to; I'm describing what I do, not what you can do :)), but rather just iterate over the underlying parquet files using filename listing or wildcarding. I often run queries on BigQuery and suck down the results to a bunch of ~1…

650GB is something one could handle using a local filesystem, no need for complex tooling.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#53

Post by tacker2000 »

gdevenyi wrote: post #21

I hate this screenshots for commands and outputs everywhere

I also hate it. Cant read it properly on mobile and cant copy it if needed.

But the worst thing is that these are not even real screenshots, the author pasted the text into some terminal window screenshot generator tool.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#55

Post by benrutter »

nikita2206 wrote: post #54

I am not in data eng, but I do occasionally query data lake at my company. Where does Snowflake stand in this? (specially looking at that Modern Data Stack image)

I beleive snowflake has its own distributed query engine, similar to say, big query.

It's a bit of a tricky comparison because snowflake, and a lot of other tools that get referred to as "modern data stack" are very vendor based. If you're using snowflake, you're probaby using it on snowflake provided architecture with a whole load of proprietary stuff. You can't "swap in" snowflake on the same hardware like you can with spark, daft, duckdb, polars etc.

That said, iirc benchmarks normally place it very similar to spark. It's distributed, so I'd be very surprised if it wasn't in the spark/daft ballpark rather than polars/duckdb.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#56

Post by throwaw12 »

nikita2206 wrote: post #54

I am not in data eng, but I do occasionally query data lake at my company. Where does Snowflake stand in this? (specially looking at that Modern Data Stack image)

I am curious as well about this, we use Snowflake, but as a software engineer I want to understand how Spark/Databricks is different, what are we missing out?

How we work with data is simple, if SQL+dashboard solves the problem then we do it in Snowflake, if we need something more advanced, then code + bunch of SQL.

Pretty sure ML engineers work in different ways, but I don't know that side well

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#57

Post by benrutter »

I love this article! But I think this insight shouldn't be surprising. Distribution always has overheads, so if you can do things on a single machine it will almost always be faster.

I think a lot of engineers expect 100 computers to be faster than 1, because of the size comparison. But we're really looking at a process here, and a process shifting data between machines will almost always have to do more stuff, and therefore be slower.

Where spark/daft are needed is if you have 1tb of data or something crazy were a single machine isn't viable. If I'm honest though, I've seen a lot of occasions where someone thinks they have that happening, and none so far where they actually do.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#58

Post by rgblambda »

mynameisash wrote: post #8
esafak wrote: post #2

If I understand correctly, polars relies on delta-rs for Delta Lake support, and that is what does not support Deletion vectors: https://github.com/delta-io/delta-rs/issues/1094 It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark.

> It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!" and "you can throw more nodes at it easy-peasy!" And yet, there are so many cases where you can just do things with better tools. Like the time a junior engineer asked for help processing 1…

I think Spark was the best tool out there when data engineering started taking off, and it just works (provided you don't have to deal with jar dependency hell) so there's not a huge incentive to move away from it.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#59

Post by justincormack »

mrlongroots wrote: post #31
luizfelberti wrote: post #6

Honestly this benchmark feels completely dominated by the instance's NIC capacity. They used a c5.4xlarge that has peak 10Gbps bandwidth, which at a constant 100% saturation would take in the ballpark of 9 minutes to load those 650GB from S3, making those 9 minutes your best case scenario for pulling the data (without even considering writing it back!) Minute differences in how these query engines schedule IO would h…

Yep I think the value of the experiment is not clear. You want to use Spark for a large dataset with multiple stages. In this case, their I/O bandwidth is 1GB/s from S3. CPU memory bandwidth is 100-200GB/s for a multi-stage job. Spark is a way to pool memory for a large dataset with multiple stages, and use cluster-internal network bandwidth to do shuffling instead of storage. Maybe when you have S3 as your backend,…

Network bandwidth is not 20x storage ant more. An SSD is around 10GB/s now, so similar to 100Gb ethernet.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#60

Post by roeja »

nikita2206 wrote: post #54

I am not in data eng, but I do occasionally query data lake at my company. Where does Snowflake stand in this? (specially looking at that Modern Data Stack image)

Snowflake has their own sql engine and is more of a serverless option. Databricks started off with spark but now also has a sql engine(optional serverless) as well, they are using spark in the article.

The delta format is Databricks lakehouse file format, snowflake uses iceberg I believe.

Both Snowflake and Databricks also provide a ton of other features like ML, Orchestration and governance. Motherduck would be the direct competitor here.

Saying that there are now extensions to query snowflake or databricks data from duckdb for simple ad hoc querying.

Duckdb is fantastic and has saved me so many times strongly recommended.

Post reply on HN