Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

61–70 of 165 posts

Re: DuckDB Internals Part 1

#61

Post by tobilg »

willtemperley wrote: post #49

The one huge caveat for anyone that cannot use dynamic linking e.g. in an AppStore context, DuckDB isn’t a great choice. It’s very hard to statically link extensions. This is where Arrow wins I think. Arrow CPP for example has very portable builds and the C interface is very usable for building bindings. DuckDB is excellent, but it’s more a black box than a library. Edit: after a conversation with a robot, it would s…

I can't confirm this, I have several instances which have statically linked extensions...

Re: DuckDB Internals Part 1

#63

Post by levanten »

mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

It depends on your task. In analytics where you need to scan lots of data points within few columns, then columnar storage is very much the best. But for transactional workloads where you have to deal with specific entities, row based would be more advantageous. There are hybrid systems that try to be both at the same time but in my experience they end not doing either very well.

Re: DuckDB Internals Part 1

#64

Post by tobyhinloopen »

mootothemax wrote: post #54
tobyhinloopen wrote: post #50

The only reason I know and use DuckDB is because my (internal, private-use-only, experimental) vibe coded projects use it a ton. I didn't pick it - LLMs did. Until this article, I wasn't aware of what it actually is capable of. Most of these projects use JSON(L) files for storage, and duckdb to process them.

If you haven’t investigated storing in parquet format - and it doesn’t break other consumers that need your jsonl formatted files - it could be worth trialling for your use case. You’ll see vastly smaller file sizes (even more so if you use zstd compression), and querying time will shoot up. Usual caveats apply, but as a general rule it’s held up well for me. Only downside is that inspecting the results moves from vi…

I'll 100% try DuckDB in more serious projects where I would normally use Sqlite.

Re: DuckDB Internals Part 1

#65

Post by raihansaputra »

jkubicek wrote: post #4
steve_adams_86 wrote: post #2

> DuckDB has received widespread adoption because it's just so damn easy to use. This was a major factor in my initial adoption. Since then it has stuck because it’s also absurdly capable, versatile, and fast. If it wasn’t so easy to use I suspect I wouldn’t have adopted it when I did. The ergonomics are crazy. It still impresses me regularly.

What do you use it for? I’m perpetually interested in using DuckDB, but it doesn’t seem to do anything I need.

throwing in my 2 cents: It just replaced pandas for me. It's just so much easier to write sql against csv/json/whatever format data in jupyter/marimo notebooks through duckdb rather than reasoning through pandas. SQL is far more natural for me, and agents also work through it easily.

Re: DuckDB Internals Part 1

#66

Post by coldtea »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

Precisely to avoid the custom NIH Polars API, and use SQL which works everywhere (yes, inconsistencies aside).

Re: DuckDB Internals Part 1

#67

Post by efromvt »

Panzerschrek wrote: post #26

If DuckDB is so fast and has no data transfer overheads, does it need all this typical SQL machinery with filtering and joining via SELECT queries? Wouldn't it be simpler and faster to return all data to the caller code (all table rows, but only requested columns) and let it perform all other necessary data processing logic?

The SELECT machinery is the product with databases! SQL often the shortest description of the processing logic, and the database has an efficient local execution engine that can prune/reduce data read based on the plan. Very hard to match in app, especially when joins get involved.

Re: DuckDB Internals Part 1

#68

Post by pknerd »

faangguyindia wrote: post #40
pknerd wrote: post #29

Earlier quoted context omitted.

Just curious whether one can earn money making these exts?

You can definately offer consultation or custom integration.

Thanks for your kind response. Could you guide further? Like businesses don't care about the tool/tech itself, how do I find and approach them, and for which niche.

Thanks in Advance

Re: DuckDB Internals Part 1

#69

Post by willtemperley »

tobilg wrote: post #61
willtemperley wrote: post #49

The one huge caveat for anyone that cannot use dynamic linking e.g. in an AppStore context, DuckDB isn’t a great choice. It’s very hard to statically link extensions. This is where Arrow wins I think. Arrow CPP for example has very portable builds and the C interface is very usable for building bindings. DuckDB is excellent, but it’s more a black box than a library. Edit: after a conversation with a robot, it would s…

I can't confirm this, I have several instances which have statically linked extensions...

I didn't say it was impossible but there are specific situations in which it's very difficult. What I was unable to resolve was getting DuckDB to statically link with httpfs and I'm not the only one [1].

So being more specific, I don't know how I could get a static build of DuckDB to work with Parquet and httpfs (i.e. query S3) working in an app store environment. It was a day's work to get Arrow CPP to call back into Swift for the transport layer.

However I do now see that DuckDB recently provided an extension point for providing your own transport layer, so my point might well be moot for that reason [2].

[1] https://github.com/duckdb/duckdb/issues/16190 [2] https://github.com/duckdb/duckdb/pull/17464

Re: DuckDB Internals Part 1

#70

Post by pimeys »

cyanide911 wrote: post #47
0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

>Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Nice! How do you set things up so that your engineers's claude code sessions upload to S3? Thanks for the help in advance

If you use OpenCode, the sessions are all in a local sqlite database. After lunch I'm pushing one of my agents to crunch some data from that using duckdb...
Post reply on HN