Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

duckdb.org

Post reply on HN
Print view View original

41–50 of 54 posts

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#41

Post by goerch »

porridgeraisin wrote: post #37

I use duckdb often too, but the way it is being hyped in these comments makes me feel like I'm missing out on some insane usecase. I basically use it to load csv, jsonl, parquet etc etc formats and do arbitrary transformations. Are people doing something else with it?

Maybe you are unconsciously doing the right thing(TM) already? So try doing it with SQLite instead :)

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#43

Post by mmarian »

yolo3000 wrote: post #38

So is DuckDb a database or a cli tool to query all sorts of file format using SQL statements? I've used it as a CLI tool, somehow don't understand the comparison to a database, which stores your data reliably, besides responding to your SQL queries.

My personal use case is a replacement for pandas for ad hoc analysis in Jupyter notebooks, which I have to do very often these days. If I had to store the data I'd pick S3+Glue+Athena.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#46

Post by simonw »

yolo3000 wrote: post #38

So is DuckDb a database or a cli tool to query all sorts of file format using SQL statements? I've used it as a CLI tool, somehow don't understand the comparison to a database, which stores your data reliably, besides responding to your SQL queries.

It can act as an in-process database, like SQLite. You can import the library directly into your code.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#47

Post by andriy_koval »

goerch wrote: post #27
andrewstuart wrote: post #25

I found it unusable due to out of memory errors with a billion row 8 column dataset. It needs manual tuning to avoid those errors and I couldn’t find the right incantation, nor should I need to - memory management is the job of the db, not me. Far too flakey for any production usage.

That sounds like a rather serious application. Did you file an issue?

I filed many issues. They were aurtoclosed after 3 months of inactivity

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#48

Post by biophysboy »

yolo3000 wrote: post #38

So is DuckDb a database or a cli tool to query all sorts of file format using SQL statements? I've used it as a CLI tool, somehow don't understand the comparison to a database, which stores your data reliably, besides responding to your SQL queries.

Along with parsing various file formats, you can create duckdb files to store tables, and make related views, schema, etc. They also have a newer ducklake tool

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#49

Post by NorthSouthNorth »

I had a project a year ago that worked with lots and lots of csv files and it was just such a lifesaver. Doing a JOIN and WHERE on two (massive) csv files and getting instant results felt like magic. Saved me a lot of time on ad hoc queries. Also Claude being good with the tool is a bonus too. Easily my favourite piece of software recently.

Re: DuckDB 1.5.2 – SQL database that runs on laptop, server, in the browser

#50

Post by datadrivenangel »

porridgeraisin wrote: post #37

I use duckdb often too, but the way it is being hyped in these comments makes me feel like I'm missing out on some insane usecase. I basically use it to load csv, jsonl, parquet etc etc formats and do arbitrary transformations. Are people doing something else with it?

loading data and doing arbitrary transformations turns out to be a huge PITA with most other tools over a long enough timescale.
Post reply on HN