Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

41–50 of 165 posts

Re: DuckDB Internals Part 1

#41

Post by porridgeraisin »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

I understand the linting aspect but not gonna lie I understood the first one immediately way more than the 2nd one due to knowing SQL well.

Re: DuckDB Internals Part 1

#42

Post by homebessguy »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

"Languages come and go, but SQL is forever"

Re: DuckDB Internals Part 1

#43

Post by aleda145 »

smithclay wrote: post #9

If you're reading this and curious: consider writing a duckdb community extension* or contributing to an existing one* duckdb is becoming a kind of data superglue between a lot of data ecosystems (GIS, observability, analytics, lakehouses, object storage, etc) that don't talk to each other typically, and it's worth checking out in 2026. * https://github.com/duckdb/extension-template * https://duckdb.org/community_ext…

I just started doing this last week!

I'm not very good at C++, but coupled with the extension template and codex I got a basic version of my extension working within an hour. Go for it!

Re: DuckDB Internals Part 1

#44

Post by zurfer »

0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).

Re: DuckDB Internals Part 1

#45

Post by brikym »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

Polars typesafe? It doesn't show you any errors until runtime right? Kusto query language is the best I've seen at type safety and I wish open source DBs would steal some ideas from it.

Re: DuckDB Internals Part 1

#47

Post by cyanide911 »

0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

>Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them

Nice! How do you set things up so that your engineers's claude code sessions upload to S3? Thanks for the help in advance

Re: DuckDB Internals Part 1

#48

Post by tomnipotent »

zurfer wrote: post #44
0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).

Not to mention it can query across heterogeneous sources, so the same query can use a duckdb table, sqlite, csv, and parquet (including predicate pushdown).

Re: DuckDB Internals Part 1

#49

Post by willtemperley »

The one huge caveat for anyone that cannot use dynamic linking e.g. in an AppStore context, DuckDB isn’t a great choice. It’s very hard to statically link extensions.

This is where Arrow wins I think. Arrow CPP for example has very portable builds and the C interface is very usable for building bindings.

DuckDB is excellent, but it’s more a black box than a library.

Edit: after a conversation with a robot, it would seem that the DuckDB and ArrowCPP C APIs are complimentary, so it's very possible to have Arrow CPP and DuckDB to coexist in an app, each with its own strength. Arrow CPP doen't have a simple SQL story for example.

Re: DuckDB Internals Part 1

#50

Post by tobyhinloopen »

The only reason I know and use DuckDB is because my (internal, private-use-only, experimental) vibe coded projects use it a ton. I didn't pick it - LLMs did. Until this article, I wasn't aware of what it actually is capable of.

Most of these projects use JSON(L) files for storage, and duckdb to process them.

Post reply on HN