Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

81–90 of 165 posts

Re: DuckDB Internals Part 1

#81

Post by pierregillesl »

pknerd wrote: post #30

umm can we say it can replace SQLite?

I wouldn’t see it as a replacement.

SQLite and DuckDB solve different problems and actually complement each other quite well. SQLite is excellent for transactional workloads (OLTP), while DuckDB shines for analytical workloads (OLAP), especially time-series data and aggregations.

We’ve been using both side-by-side in an open-source project for about two years: SQLite for configuration and transactional data, DuckDB for historical sensor data and analytics. So far, it’s been a very good combination.

Re: DuckDB Internals Part 1

#82

Post by medvezhenok »

mootothemax wrote: post #54
tobyhinloopen wrote: post #50

The only reason I know and use DuckDB is because my (internal, private-use-only, experimental) vibe coded projects use it a ton. I didn't pick it - LLMs did. Until this article, I wasn't aware of what it actually is capable of. Most of these projects use JSON(L) files for storage, and duckdb to process them.

If you haven’t investigated storing in parquet format - and it doesn’t break other consumers that need your jsonl formatted files - it could be worth trialling for your use case. You’ll see vastly smaller file sizes (even more so if you use zstd compression), and querying time will shoot up. Usual caveats apply, but as a general rule it’s held up well for me. Only downside is that inspecting the results moves from vi…

Yup, especially data backups (although I wouldn't store critical backups like this, only nice-to-have ones). One minor note is that parquet file sizes / compressed sizes can be sensitive to ordering, so you can try different sort orders to get optimal compression.

I found with using various tricks I can get the zstd parquet to be up to 10x (or more) smaller than an equivalent Postgres table - but obviously the exact compression ratios will depend on the kind of data you have and how well your Postgres table is normalized.

Re: DuckDB Internals Part 1

#83

Post by axegon_ »

I use duckdb HEAVILY at work and it's been a game changer. I'm sifting through terabytes of data multiple times a day, mixing, matching, updating, filtering, DuckDB is second to none. For anyone that hasn't used it: you are missing out.

Re: DuckDB Internals Part 1

#84

Post by brandmeyer »

levanten wrote: post #63
mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

It depends on your task. In analytics where you need to scan lots of data points within few columns, then columnar storage is very much the best. But for transactional workloads where you have to deal with specific entities, row based would be more advantageous. There are hybrid systems that try to be both at the same time but in my experience they end not doing either very well.

Some day we'll get CREATE TABLE ... ( ... STORAGE ORDER COLUMN MAJOR) to have our transactional cake on the tables that need it and eat our analytics cake on the tables that need that.

But until then, separate tools for separate purposes isn't a bad place to be when those tools are both fantastic.

Re: DuckDB Internals Part 1

#86

Post by tosh »

0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

Agree, in addition to that DuckDB also works quite well for data that is too big to fit in memory or on the machine DuckDB is on (predicate push down, out of core processing, …).

Re: DuckDB Internals Part 1

#87

Post by _boffin_ »

cyanide911 wrote: post #47
0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

>Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Nice! How do you set things up so that your engineers's claude code sessions upload to S3? Thanks for the help in advance

Probably on a business / Enterprise plan, which has managed settings and also telemetry export. Give it a collector endpoint to export to and then have collector send to s3.

Re: DuckDB Internals Part 1

#88

Post by sigbottle »

What the fuck.

I've never been that strong of an engineer. TIL that at one of my internships I was building DuckDB but for the company's private use cases. Well, trying to anyways. I didn't really get the whole picture, the pieces did not fit into place.

Didn't get the return offer obviously, probably because I didn't make the connection (or really a coherent narrative of what I was building). RIP. You live and you learn, I guess.

Re: DuckDB Internals Part 1

#89

Post by vanyaland »

smithclay wrote: post #9

If you're reading this and curious: consider writing a duckdb community extension* or contributing to an existing one* duckdb is becoming a kind of data superglue between a lot of data ecosystems (GIS, observability, analytics, lakehouses, object storage, etc) that don't talk to each other typically, and it's worth checking out in 2026. * https://github.com/duckdb/extension-template * https://duckdb.org/community_ext…

[flagged]

Re: DuckDB Internals Part 1

#90

Post by ryanchants »

ashu1461 wrote: post #71
0xferruccio wrote: post #17

DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…

Can you please expand more on the claude analysis part. What exactly you analysed and what outcome it helped with ?

Not who you responded to, but I've been working on cctx. It's an open source tool for analyzing claude code sessions to see where things went wrong(tool failure loops, bloated context, and the like).

https://github.com/jacquardlabs/cctx

Post reply on HN