Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Internals Part 1

greybeam.ai

Post reply on HN
Print view View original

121–130 of 165 posts

Re: DuckDB Internals Part 1

#121

Post by kristjansson »

pants2 wrote: post #120
xtracto wrote: post #116

Earlier quoted context omitted.

This may be useful for somebody: We are also using DuckDB heavily at my workplace (we do Tax analytics of very large companies with huge amounts of data). We have certain DuckDB processes that happened in AWS infrastructure, where the data is saved in GP3 disks. We didn't know that for GP3 disks, you can increase not only IOPS but also Read/Write Throughput [1] which by default is 125 MB/s. So by default we were not…

This seems crazy low to me. AWS has default 3K IOPS and 125 MB/s throughput, meanwhile my Macbook Pro has 700K IOPS and 14.5GB/s throughput. Is Amazon running on super outdated legacy networking?

SAN vs local. Local NVME (“instance storage“) on AWS is wicked fast too, but live and dies with the instance

Re: DuckDB Internals Part 1

#122

Post by kristjansson »

ndr wrote: post #56
zurfer wrote: post #44

Earlier quoted context omitted.

Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).

This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…

> no data hallucinations

Dangerous thing to assert. It’ll happily run SQL that works, but doesn’t necessarily correspond to intentions or unstated assumptions about the data.

Re: DuckDB Internals Part 1

#123

Post by kristjansson »

noworriesnate wrote: post #109
ndr wrote: post #56

Earlier quoted context omitted.

This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…

I have a a theory that LLMs are going to be the death knell of big SaaS. It's so much harder to build and maintain an massive SaaS that does 80% of what 80% of your customers want, than it is to build something small and simple that does 100% of what one customer wants.

Maybe once the model can administer and operate the service too.

For now building the 10% of the SaaS that you need still leaves you operating 100% of a new service/process

Re: DuckDB Internals Part 1

#124

Post by ndr »

kristjansson wrote: post #122
ndr wrote: post #56

Earlier quoted context omitted.

This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…

> no data hallucinations Dangerous thing to assert. It’ll happily run SQL that works, but doesn’t necessarily correspond to intentions or unstated assumptions about the data.

Of course I meant that it won't make data up.

It can only emit SQL and the json spec of the chart.

Since shipping I've reviewed dozens of queries and charts it produces answering the user. I'm yet to catch sonnet off guard.

Re: DuckDB Internals Part 1

#125

Post by charlieflowers »

mcv wrote: post #59

Is everything becoming columnar? Parquet stores data per column instead of per row because it improves compression. I get that. Arrow apparently is columnar, and now DuckDB also gets its efficiency by treating data as columns instead of rows? I still need to wrap my head around how that works, but it's a fascinating development.

BTW, columnar is very similar to struct of arrays (SOA) and some of the reasons it works well overlap with SOA.

Re: DuckDB Internals Part 1

#127

Post by xearl »

romaniv wrote: post #91

It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…

> while DuckDB effectively appeared on the scene two years ago.

duckdb is ~7 years old by now. it was quite popular long before it became 1.0. heck, even motherduck has been founded 4 years ago.

Re: DuckDB Internals Part 1

#130

Post by SmirkingRevenge »

refactor_master wrote: post #39
holografix wrote: post #24

Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?

The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…

Polars and duckdb interoperate so it's not really one or the other.

Other dataframe libraries work directly on duckdb also, like Ibis.

FWIW, I think SQLAlchemy does also.

In any case though, I find both the SQL and the code there equally readable.

Post reply on HN