Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Untitled topic

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 49 posts

Re: undefined

#21

Post by jitl »

chatmasta wrote: post #12
netcraft wrote: post #11

Why would this be useful over of DuckDb? (earnest question)

They’re similar, but DuckDb is more of a batteries-included database whereas DataFusion is an embeddable query engine. You can use DuckDb in embedded-ish scenarios, but it’s not primarily targeting that use case. To put it another way, DataFusion is sometimes described as “the LLVM of databases.” Another difference is that DuckDb is written in C++ whereas DataFusion is in Rust, so all the usual memory-safety and perf…

We tried both about 8 months ago, at the time DuckDB’s Node driver leaked memory and segfaulted, DataFusion was missing some features we wanted. But they are both improving rapidly.

Re: undefined

#22

Post by [deleted] »

lidavidm wrote: post #10
elasticventures wrote: post #7

Earlier quoted context omitted.

Exactly, datafusion is implied batteries included apache bigdata ecosystem. Polars is chasing the Python Pandas crowd and uses python syntax, handy if you're already comfortable with ipython.

Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...) I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars…

[deleted]

Re: undefined

#23

Post by geysersam »

chatmasta wrote: post #12
netcraft wrote: post #11

Why would this be useful over of DuckDb? (earnest question)

They’re similar, but DuckDb is more of a batteries-included database whereas DataFusion is an embeddable query engine. You can use DuckDb in embedded-ish scenarios, but it’s not primarily targeting that use case. To put it another way, DataFusion is sometimes described as “the LLVM of databases.” Another difference is that DuckDb is written in C++ whereas DataFusion is in Rust, so all the usual memory-safety and perf…

> DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year

Really? I don't see it near the top.

[CH benchmarks](https://benchmark.clickhouse.com/#eyjzexn0zw0ionsiqwxsb3leqi...)

Re: undefined

#24

Post by Epa095 »

lidavidm wrote: post #10
elasticventures wrote: post #7

Earlier quoted context omitted.

Exactly, datafusion is implied batteries included apache bigdata ecosystem. Polars is chasing the Python Pandas crowd and uses python syntax, handy if you're already comfortable with ipython.

Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...) I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars…

"pg_analytics (formerly named pg_lakehouse) puts DuckDB inside Postgres" https://github.com/paradedb/pg_analytics

Re: undefined

#25

Post by hipadev23 »

jitl wrote: post #20
jamesblonde wrote: post #9

There is a cambrian explosion in data processing engines: DataFusion, Polars, DuckDB, Feldera, Pathway, and more than i can remember. It reminds of 15 years ago where there was JDBC/ODBC for data. Then when data volumes increased, specialized databases became viable - graph, document, json, key-value, etc. I don't see SQL and Spark hammers keeping their ETL monopolies for much longer.

Spark for sure I view with suspicion and avoid as much as possible at work. SQL though is going the distance. like Feldera is SQL based stream processing and uses DataFusion under the hood for some data wrangling. DuckDB is also very SQL. I have my quibbles with SQL as a language but I would prefer SQL embedded in $myLanguage to needing to use Python or (shudder) Scala to screw around with data.

Absolutely agree. Spark is the same garbage as Hadoop but in-memory.

Re: undefined

#26

Post by lidavidm »

Epa095 wrote: post #24
lidavidm wrote: post #10

Earlier quoted context omitted.

Can't you use DataFusion single node/without any Apache ecosystem stuff? They have a Python library and DataFusion is "just" a query engine. (If anything, I'd call Pandas the batteries included option...) I think the difference is more that DataFusion is built as a library so you can plug it into the product you're building (e.g. Comet, which plugs it into Spark, or pg_lakehouse, which plugs it into Postgres). Polars…

"pg_analytics (formerly named pg_lakehouse) puts DuckDB inside Postgres" https://github.com/paradedb/pg_analytics

It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse

Re: undefined

#27

Post by RobinL »

krapht wrote: post #2

I feel like I'm not the target audience for this. When I have large data, then I directly write SQL queries and run them against the database. It's impossible to improve performance when you have to go out to the DB anyway; might as well have it run the query too. Certainly the server ops and db admins have loads more money to spend on making the DB fast compared with my anti-virus laden corporate laptop. When I have…

> It's impossible to improve performance when you have to go out to the DB anyway;

That's not right. There are many queries that run far faster in duckdb/datafusion than (say) postgres, even with the overhead of pulling whole large tables prior to running the query. (Or use like pg_duckdb).

For certain types of queries these engines can be 100x faster.

More here: https://postgres.fm/episodes/pg_duckdb

Re: undefined

#28

Post by jurgenaut23 »

jamesblonde wrote: post #9

There is a cambrian explosion in data processing engines: DataFusion, Polars, DuckDB, Feldera, Pathway, and more than i can remember. It reminds of 15 years ago where there was JDBC/ODBC for data. Then when data volumes increased, specialized databases became viable - graph, document, json, key-value, etc. I don't see SQL and Spark hammers keeping their ETL monopolies for much longer.

Did you really put SQL and Spark in the same basket?

Re: undefined

#29

Post by GardenLetter27 »

threeseed wrote: post #8
krapht wrote: post #2

I feel like I'm not the target audience for this. When I have large data, then I directly write SQL queries and run them against the database. It's impossible to improve performance when you have to go out to the DB anyway; might as well have it run the query too. Certainly the server ops and db admins have loads more money to spend on making the DB fast compared with my anti-virus laden corporate laptop. When I have…

You’re right it isn’t for you. It’s largely for companies who can’t put everything in a single database because (a) they don’t control the source schema e.g. it’s a daily export from a SaaS app, (b) the ROI is not high enough to do so and (c) it’s not in a relational format e.g. JSON, Logs, Telemetry etc. And with the trend toward SaaS apps it’s a situation that is becoming more common.

Or when the data is massive - so even BigQuery would be crazy expensive.

Re: undefined

#30

Post by kalendos »

geysersam wrote: post #23
chatmasta wrote: post #12

Earlier quoted context omitted.

They’re similar, but DuckDb is more of a batteries-included database whereas DataFusion is an embeddable query engine. You can use DuckDb in embedded-ish scenarios, but it’s not primarily targeting that use case. To put it another way, DataFusion is sometimes described as “the LLVM of databases.” Another difference is that DuckDb is written in C++ whereas DataFusion is in Rust, so all the usual memory-safety and perf…

> DataFusion has recently overtaken DuckDb in Clickbench results after a community push last year Really? I don't see it near the top. [CH benchmarks]( https://benchmark.clickhouse.com/#eyjzexn0zw0ionsiqwxsb3leqi... )

You might need to adjust filters to do an apple to apple comparison.

https://benchmark.clickhouse.com/#eyJzeXN0ZW0iOnsiQWxsb3lEQi...

Post reply on HN