Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Untitled topic

news.ycombinator.com

Post reply on HN
Print view View original

41–49 of 49 posts

Re: undefined

#41

Post by alamb »

netcraft wrote: post #11

Why would this be useful over of DuckDb? (earnest question)

I think you would pick DataFusion over DuckDB if you want to customize it substantially. Not just with user defined functions (which are quite easy to write in DataFusion and are very fast), but things like * custom file formats (e.g. Spiral or Lance) * custom query languages / sql dialects * custom catalogs (e.g. other than a local file or prebuilt duckdb connectors) * custom indexes (read only parts of parquet files based on extra information you store) * etc.

If you are looking for the nicest "run SQL on local files" experience, DuckDB is pretty hard to beat

Disclaimer: I am the PMC chair of DataFusion

There are some other interesting FAQs here too: https://datafusion.apache.org/user-guide/faq.html

Re: undefined

#42

Post by alamb »

geysersam wrote: post #23
chatmasta wrote: post #12

Earlier quoted context omitted.

They’re similar, but DuckDb is more of a batteries-included database whereas DataFusion is an embeddable query engine. You can use DuckDb in embedded-ish scenarios, but it’s not primarily targeting that use case. To put it another way, DataFusion is sometimes described as “the LLVM of databases.” Another difference is that DuckDb is written in C++ whereas DataFusion is in Rust, so all the usual memory-safety and perf…

> DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year Really? I don't see it near the top. [CH benchmarks]( https://benchmark.clickhouse.com/#eyjzexn0zw0ionsiqwxsb3leqi... )

Specifically, DataFusion is faster when querying parquet directly.

Most of the leaderboard of ClickBench is for database specific file formats (that you first have to load the data into)

Re: undefined

#43

Post by philippemnoel »

lidavidm wrote: post #26
Epa095 wrote: post #24

Earlier quoted context omitted.

"pg_analytics (formerly named pg_lakehouse) puts DuckDB inside Postgres" https://github.com/paradedb/pg_analytics

It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse

That's true. We have some more ideas for DataFusion in the works, though... Stay tuned!

Re: undefined

#44

Post by riku_iki »

kalendos wrote: post #30
geysersam wrote: post #23

Earlier quoted context omitted.

> DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year Really? I don't see it near the top. [CH benchmarks]( https://benchmark.clickhouse.com/#eyjzexn0zw0ionsiqwxsb3leqi... )

You might need to adjust filters to do an apple to apple comparison. https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

Not clear why someone need to give up on native duckdb format if it is much faster.

Re: undefined

#45

Post by chatmasta »

riku_iki wrote: post #44
kalendos wrote: post #30

Earlier quoted context omitted.

You might need to adjust filters to do an apple to apple comparison. https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

Not clear why someone need to give up on native duckdb format if it is much faster.

Because it means you need to keep another copy of your data in a special format just for DuckDb. The point of Parquet is that it’s an open format queryable by multiple tools. You don’t need to wait to load every table into a new format, you don’t need to retain multiple copies, and you don’t need to keep them in sync.

If DuckDb is the only query engine in your analytics stack, then it makes sense to use its specialized format. But that’s not the typical Lakehouse use case.

Re: undefined

#46

Post by riku_iki »

chatmasta wrote: post #45
riku_iki wrote: post #44

Earlier quoted context omitted.

Not clear why someone need to give up on native duckdb format if it is much faster.

Because it means you need to keep another copy of your data in a special format just for DuckDb. The point of Parquet is that it’s an open format queryable by multiple tools. You don’t need to wait to load every table into a new format, you don’t need to retain multiple copies, and you don’t need to keep them in sync. If DuckDb is the only query engine in your analytics stack, then it makes sense to use its specializ…

> But that’s not the typical Lakehouse use case.

that benchmark is also not typical lakehouse use case, since all data is hosted locally, so they don't test significant component of the stack.

Re: undefined

#47

Post by chatmasta »

riku_iki wrote: post #46
chatmasta wrote: post #45

Earlier quoted context omitted.

Because it means you need to keep another copy of your data in a special format just for DuckDb. The point of Parquet is that it’s an open format queryable by multiple tools. You don’t need to wait to load every table into a new format, you don’t need to retain multiple copies, and you don’t need to keep them in sync. If DuckDb is the only query engine in your analytics stack, then it makes sense to use its specializ…

> But that’s not the typical Lakehouse use case. that benchmark is also not typical lakehouse use case, since all data is hosted locally, so they don't test significant component of the stack.

Yeah, that’s one of many issues with Clickbench. It’s also one table so it can’t test joins.

TPC-H is okay but not Lakehouse specific. I’m not aware of any benchmarks that specifically test performance of engines under common setups like external storage or scalable compute. It would be hard to design one that’s easily reproducible. (And in fairness to Clickbench, it’s intentionally simple for that exact reason - to generate a baseline score for any query engine that can query tabular data).

Re: undefined

#48

Post by theLiminator »

I've done some testing of polars, duckdb, and datafusion.

Anecdotally, these are my experiences:

DuckDB (last used maybe 7-8 months):

- Very nice for very fast local queries (against parquet files, i ignored their homegrown file format)

- Most pleasant cli

- Seems to have the best out of core experience

- As far as I can tell, seems to be closest to state of the art in terms of algorithms/overall design, though honestly everyone is within spitting distance of each other

- Spark api seems exciting

Datafusion (last used 1.5y ago):

- Most pleasant to build/extend on top of (in rust)

- Is to OLAP DBMS's what LLVM is to compilers (stole this quote off Andrew Lamb)

- Could be wrong, but in terms of core engineering discipline they are the most rigorous/thoughtful (no shade thrown to the other libraries, which are all awesome libraries/tools too)

- Seems to be the most foundational to many other tools (and is most ubiquitously embedded)

- Their python dataframe centric workflow isn't as nice as polars (this is rapidly improving afaict)

- Docs are lagging behind polars

- Very exciting future (ray datafusion, improvements to python bindings, ballista, datafusion-comet)

Polars (last used this week):

- The most pleasant api by far for a programmatic user

- Pretty good interop with python ecosystem

- Rust crate is a second class citizen

- Python is a first class citizen

- Probably the best for advanced ETL use cases

- Fastest library for querying hive partitioned parquet data in an object store

- Wide end-user adoption (less so as a query engine)

- Moves very fast (I do get more bugs/regressions in polars version to version, but on the flip side, they move fast to fix issues and release very often)

- Exciting distributed cloud solution coming (is proprietary though)

- New streaming engine based off morsel driven parallelism (same architectural as duckdb afaict?) should greatly improve polars OOC capabilities

- Much nicer to test/compose/build re-usable queries/functions on top of then SQL based ETL tools - Error messages/debuggability/observability are still immature

All three are awesome tools. The OLAP space is really heating up.

Things I still see lacking in the OLAP end-user space are: - Unified batch/streaming dataframe centric workflows, nothing is truly high throughput/low latency/pleasant to use/mature/robust. I've only really seen arroyo and risingwave, neither seem too mature usable yet.

- Nothing is quite at the robustness level of something like sqlite

- Despite native query engines, datalake implementations are mostly lagging behind their java equivalents (iceberg/delta)

Some questions for other users:

- I'm curious if anyone uses Ibis in prod, I found that it wasn't very usable as an end user

Re: undefined

#49

Post by hipadev23 »

ignoreusernames wrote: post #32
hipadev23 wrote: post #25

Earlier quoted context omitted.

Absolutely agree. Spark is the same garbage as Hadoop but in-memory.

just out of curiosity, why do you say that spark is "in-memory"? I see a lot people claiming that, including several that I've interviewed in the past few years but that's not very accurate(at least in the default case). Spark SQL execution uses a bog standard volcano-ish iterator model (with a pretty shitty codegen operator merging part) built on top of their RDD engine. The exchange (shuffle) is disk based by defau…

Because that was the central point in the original whitepaper [1]: Hadoop is slow because it’s disk-only where Spark uses memory and caching to speed things up. I understand Spark isn’t 100% in-memory the way say Redis is, but it was still the major selling point vs. Hadoop.

https://people.csail.mit.edu/matei/papers/2010/hotcloud_spar...

Post reply on HN