Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

dataengineeringcentral.substack.com

Post reply on HN
Print view View original

61–70 of 112 posts

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#61

Post by prpl »

roeja wrote: post #60
nikita2206 wrote: post #54

I am not in data eng, but I do occasionally query data lake at my company. Where does Snowflake stand in this? (specially looking at that Modern Data Stack image)

Snowflake has their own sql engine and is more of a serverless option. Databricks started off with spark but now also has a sql engine(optional serverless) as well, they are using spark in the article. The delta format is Databricks lakehouse file format, snowflake uses iceberg I believe. Both Snowflake and Databricks also provide a ton of other features like ML, Orchestration and governance. Motherduck would be the…

"but now also has a sql engine"

it has had a combined SQL and dataframe engine since March 2015...

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#64

Post by datacynic »

garganzol wrote: post #18
jdnier wrote: post #7

DuckDb has a new "DuckLake" catalog format that would be another candidate to test. https://ducklake.select/

DuckLake format has an unresolved built-in chicken and egg conflict: it requires SQL database to represent its catalog. But this is what some people are running away from when they choose Parquet format in the first place. Parquet = easy, SQL = hard, adding SQL to Parquet makes the resulting format hard. I would expect a catalog to be in Parquet format as well, then it becomes something self-bootstrapping and usable.

DuckLake is more comparable to Iceberg and Delta than to raw parquet files. Iceberg requires a catalog layer too, a file system based one at its simplest. For DuckLake any RDBMS will do, including fs-based ones like DuckDB and SQLite. The difference is that DuckLake will use that database with all its ACID goodness for all metadata operations and there is no need to implement transactional semantics over a REST or object storage API.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#65

Post by l_c_m »

This is most misrepresented article on two fronts

1. tested column pruning and the dataset you access would have been 2 columns + metadata for the parquet files so probably fit in memory even without streaming.

2. Most of the processing time would be IO bound on S3 and the access patterns/simultaneous connection limits etc. would have more of an impact than any processing code.

Love that you went through the pain of trying the different systems but I'd like to see an actual larger than memory query.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#68

Post by benrutter »

rgblambda wrote: post #58
mynameisash wrote: post #8

Earlier quoted context omitted.

> It seems like these single-node libraries can process a terabyte on a typical machine, and you'd have have over 10TB before moving to Spark. I'm surprised by how often people jump to Spark because "it's (highly) parallelizable!" and "you can throw more nodes at it easy-peasy!" And yet, there are so many cases where you can just do things with better tools. Like the time a junior engineer asked for help processing 1…

I think Spark was the best tool out there when data engineering started taking off, and it just works (provided you don't have to deal with jar dependency hell) so there's not a huge incentive to move away from it.

This is so true! Even a few years ago, these benchmarks would have been against pandas (instead of polaes and duckdb) and would likely have looked very different.

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#69

Post by capitol_ »

miohtama wrote: post #52
willvarfar wrote: post #39

I often crunch 'biggish data' on a single node using duckdb (because I love using the modern style of painless and efficient SQL engines). I don't use delta or iceberg (because I haven't needed to; I'm describing what I do, not what you can do :)), but rather just iterate over the underlying parquet files using filename listing or wildcarding. I often run queries on BigQuery and suck down the results to a bunch of ~1…

650GB is something one could handle using a local filesystem, no need for complex tooling.

650 GB fits in ram: https://yourdatafitsinram.net

Re: 650GB of Data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark

#70

Post by data_marsupial »

pu_pe wrote: post #45

In places I have worked at that used Databricks, I feel they chose it for the same reasons big orgs use Microsoft: it comes out of a box and has a big company behind it. Technical benchmarks or even cost considerations would be a distant second.

There are real advantages from having a managed data platform compared to managing everything yourself, especially if you have a large number of data teams that need to collaborate.
Post reply on HN