Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why DuckDB is my first choice for data processing

robinlinacre.com

Post reply on HN
Print view View original

61–70 of 124 posts

Re: Why DuckDB is my first choice for data processing

#61

Post by willvarfar »

pattar wrote: post #26
biophysboy wrote: post #14

Earlier quoted context omitted.

DuckLake is pretty new, so I guess it would depend on if you need a more mature, fully-featured app.

I went to a talk by the Motherduck team about why they built DuckLake instead of leaning more in on Iceberg. The key takeaway is that instead of storing all the table metadata inside files on s3 and dealing with latency and file io they instead store all of that info inside a duckdb table. Seems like a good idea and worked smoothly when I tried, however it is not quite in a stable production state it is still https:/…

(I work a lot with BigQuery's BigLake adaptor and it's basically caching the metadata of the iceberg manifest and parquet footers in Bigtable (this is Google) so query planning is super fast etc. Really helps)

Re: Why DuckDB is my first choice for data processing

#62

Post by yakkomajuri »

yakkomajuri wrote: post #56

Been quite a fan of DuckDB and we actually even use it in production. But coincidentally today I was exploring memory usage and I believe I'm finding memory leaks. Anybody have similar experiences? Still debugging more deeply but looking reasonably conclusive atm.

As far as I can tell pretty conclusive results:

https://github.com/duckdb/duckdb/issues/20569

If someone can explain this or has a fix for it I'd love to hear it!

Re: Why DuckDB is my first choice for data processing

#63

Post by skeeter2020 »

mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

>> The .csv parser is amazing

Their csv support coupled with lots of functions and fast & easy iterative data discovery has totally changed how I approach investigation problems. I used to focus a significant amount of time on understanding the underlying schema of the problem space first, and often there really wasn't one - but you didn't find out easily. Now I start with pulling in data, writing exploratory queries to validate my assumptions, then cleaning & transforming data and creating new tables from that state; rinse and repeat. Aside from getting much deeper much quicker, you also hit dead ends sooner, saving a lot of otherwise wasted time.

There's an interesting paper out there on how the CSV parser works, and some ideas for future enhancements. I couldn't seem to find it but maybe someone else can?

Re: Why DuckDB is my first choice for data processing

#64

Post by udkl »

uwemaurer wrote: post #50

We use DuckDB to process analytics and feeds for Bluesky ( https://bluefacts.app ) To get fast access to the query results we use the Apache Arrow interface and generate the code directly from DuckDB SQL queries using the SQG tool ( https://sqg.dev/generators/java-duckdb-arrow/ )

I'm very interested to see a behind the scenes of your tech - do you guys have a writeup somewhere? You HN tool is also promising!

Re: Why DuckDB is my first choice for data processing

#66

Post by tosh »

skeeter2020 wrote: post #63
mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

>> The .csv parser is amazing Their csv support coupled with lots of functions and fast & easy iterative data discovery has totally changed how I approach investigation problems. I used to focus a significant amount of time on understanding the underlying schema of the problem space first, and often there really wasn't one - but you didn't find out easily. Now I start with pulling in data, writing exploratory queries…

not a paper but I found this: https://duckdb.org/2025/04/16/duckdb-csv-pollock-benchmark

Re: Why DuckDB is my first choice for data processing

#67

Post by dkdcio »

red2awn wrote: post #65

It was mentioned that the performance of DuckDB is similar to that of Polars (among others). In that case why would one choose DuckDB over Polars? The only differentiator seems to be that you do the querying with standard SQL instead of the library specific APIs.

- performance is often better, especially on “out of core” (“streaming”, spill to disk data sizes). Polars has done a ton of work on their streaming engine but they’re still catching up

- you don’t need to use Python (but Pythonic wrappers like Ibis exist; disclaimer I worked on Ibis, you can find my blogs on performance comparisons easily there); CLI, WASM, etc. w/o Python

- governance: DuckDB as OSS is setup in a more sustainable way (DuckDB Labs + DuckDB Foundation). while there is a VC-backed company (MotherDuck), it doesn’t employ the primary developers/control the project in the same way the Polars company does

- overall just simplicity and focus. tends to break less, solely focused on single-node, easy to extend, etc. — not trying to do a cloud product, distributing computing, supporting GPU execution

Re: Why DuckDB is my first choice for data processing

#68

Post by hk1337 »

mrtimo wrote: post #19

What I love about duckdb: -- Support for .parquet, .json, .csv (note: Spotify listening history comes in a multiple .json files, something fun to play with). -- Support for glob reading, like: select * from 'tsa20*.csv' - so you can read hundreds of files (any type of file!) as if they were one file. -- if the files don't have the same schema, union_by_name is amazing. -- The .csv parser is amazing. Auto assigns type…

-- hive partitioning

Re: Why DuckDB is my first choice for data processing

#69

Post by physicsguy »

noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

You can get 32TiB of RAM instances on AWS these days

Re: Why DuckDB is my first choice for data processing

#70

Post by RobinL »

theLiminator wrote: post #51
noo_u wrote: post #15

I'd say the author's thoughts are valid for basic data processing. Outside of that, most of claims in this article, such as: "We're moving towards a simpler world where most tabular data can be processed on a single large machine1 and the era of clusters is coming to an end for all but the largest datasets." become very debatable. Depending on how you want to pivot/ scale/augment your data, even datasets that seeming…

Yeah, i'm also similarly confused. > "SQL should be the first option considered for new data engineering work. It’s robust, fast, future-proof and testable. With a bit of care, it’s clear and readable." (over polars/pandas etc) SQL has nothing to do with fast. Not sure what makes it any more testable than polars? Future-proof in what way? I guess they mean your SQL dialect won't have breaking changes?

Author here. I wouldn't argue SQL or duckdb is _more_ testable than polars. But I think historically people have criticised SQL as being hard to test. Duckdb changes that.

I disagree that SQL has nothing to do with fast. One of the most amazing things to me about SQL is that, since it's declarative, the same code has got faster and faster to execute as we've gone through better and better SQL engines. I've seen this through the past five years of writing and maintaining a record linkage library. It generates SQL that can be executed against multiple backends. My library gets faster and faster year after year without me having to do anything, due to improvements in the SQL backends that handle things like vectorisation and parallelization for me. I imagine if I were to try and program the routines by hand, it would be significantly slower since so much work has gone into optimising SQL engines.

In terms of future proof - yes in the sense that the code will still be easy to run in 20 years time.

Post reply on HN