Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

duckdb.org

Post reply on HN
Print view View original

31–40 of 54 posts

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#31

Post by andrewstuart »

goerch wrote: post #29
andrewstuart wrote: post #28

Earlier quoted context omitted.

No, I tried Clickhouse instead, which worked without crashing or manual memory tuning. Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”.

Understood: SQLite is to Postgres as DuckDB is to ClickHouse.

I don’t see the analogy, if you’re using it to excuse crashing on small data sets and indexes.

SQLite isn’t small and crashy, it’s small and reliable.

There’s something fundamentally wrong with the codebase/architecture if there’s so many memory problems.

And the absolute baseline requirement for a production database is no crashes.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#32

Post by goerch »

andrewstuart wrote: post #28
goerch wrote: post #27

Earlier quoted context omitted.

That sounds like a rather serious application. Did you file an issue?

No, I tried Clickhouse instead, which worked without crashing or manual memory tuning. Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”.

> Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”.

Ah, missed this the first time around. Will check this out. And yes, I noticed that DuckDB rather aggressively tries to use the resources of your computer.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#33

Post by steve_adams_86 »

mmarian wrote: post #30

It's been a lifesaver for some analysis I had to do on 70GB of Cloudflare logs.

I'd be lost without it for log analysis. It's like a swiss army knife for making sense of disparate or large sets of data. So easy to pull up, so cooperative with data that's easy to compose from curl and bash, etc. It makes life so much easier

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#34

Post by mmarian »

steve_adams_86 wrote: post #33
mmarian wrote: post #30

It's been a lifesaver for some analysis I had to do on 70GB of Cloudflare logs.

I'd be lost without it for log analysis. It's like a swiss army knife for making sense of disparate or large sets of data. So easy to pull up, so cooperative with data that's easy to compose from curl and bash, etc. It makes life so much easier

Yep. And easy to reuse as well since it's just SQL.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#35

Post by uniqueuid »

esafak wrote: post #3

Any opinions on DuckLake?

The problem space that ducklake solves is smaller, but it helped me to get a working metabase dashboard quickly on ~1tb of data with 128gb ram. Queries were much, much faster than all alternatives.

Some downsides are: No unique constraints with indexes (can accidentally shoot yourself in the foot with double ingestion), writing is a bit cumbersome if you already have parquet files.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#36

Post by goerch »

andrewstuart wrote: post #31
goerch wrote: post #29

Earlier quoted context omitted.

Understood: SQLite is to Postgres as DuckDB is to ClickHouse.

I don’t see the analogy, if you’re using it to excuse crashing on small data sets and indexes. SQLite isn’t small and crashy, it’s small and reliable. There’s something fundamentally wrong with the codebase/architecture if there’s so many memory problems. And the absolute baseline requirement for a production database is no crashes.

Agree with your assessment of small and reliable for SQLite. Disagree with your baseline requirement. ACID is more important for me and does not contain `No crashes`.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#37

Post by porridgeraisin »

I use duckdb often too, but the way it is being hyped in these comments makes me feel like I'm missing out on some insane usecase.

I basically use it to load csv, jsonl, parquet etc etc formats and do arbitrary transformations. Are people doing something else with it?

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#38

Post by yolo3000 »

So is DuckDb a database or a cli tool to query all sorts of file format using SQL statements? I've used it as a CLI tool, somehow don't understand the comparison to a database, which stores your data reliably, besides responding to your SQL queries.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#39

Post by ciupicri »

andrewstuart wrote: post #31
goerch wrote: post #29

Earlier quoted context omitted.

Understood: SQLite is to Postgres as DuckDB is to ClickHouse.

I don’t see the analogy, if you’re using it to excuse crashing on small data sets and indexes. SQLite isn’t small and crashy, it’s small and reliable. There’s something fundamentally wrong with the codebase/architecture if there’s so many memory problems. And the absolute baseline requirement for a production database is no crashes.

I think the authors disagree with me, but I see it like a online analytical processing (OLAP) database, not like a OLTP (online transaction processing) database, so crashes are more tolerable.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#40

Post by mmarian »

porridgeraisin wrote: post #37

I use duckdb often too, but the way it is being hyped in these comments makes me feel like I'm missing out on some insane usecase. I basically use it to load csv, jsonl, parquet etc etc formats and do arbitrary transformations. Are people doing something else with it?

Probably because you probably don't have to do those arbitrary transformations that often. I do, being in a security-related role. But I wouldn't have recognized its usefulness in my previous roles as a front/backend dev.
Post reply on HN