Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The DuckDB Local UI

duckdb.org

Post reply on HN
Print view View original

111–120 of 197 posts

Re: The DuckDB Local UI

#111

Post by proamdev123 »

drdaeman wrote: post #103
gnulinux wrote: post #48

Earlier quoted context omitted.

DuckDB is mind blowingly awesome. It is like SQLite, lightweight, embeddable, serverless, in-memory database, but it's optimized to be columnar (analytics optimized). It can work with files that are in filesystem, S3 etc without copying (it just looks at the necessary regions in the file) by just doing `select * from 's3://....something.parquet'`. It support automatic compression and automatic indexing. It can read j…

I'm sorry, I must be exceptionally stupid (or haven't seriously worked in this particular problem domain and thus lacking awareness), but I still can't figure out the use cases from this feature list. What sort of thing should I be working on, to think "oh, maybe I want this DuckDB thing here to do this for me?" I guess I don't really get the "that you want to learn something about" bit.

I’m not the person you asked, but here are some random, assorted examples of “structured data you want to learn something about”:

- data you’ve pulled from an API, such as stock history or weather data,

- banking records you want to analyze for patterns, trends, unauthorized transactions, etc

- your personal fitness data, such as workouts, distance, pace, etc

- your personal sleep patterns (data retrieved from a sleep tracking device),

- data you’ve pulled from an enterprise database at work — could be financial data, transactions, inventory, transit times, or anything else stored there that you might need to pull and analyze.

Here’s a personal example: I recently downloaded a publicly available dataset that came in the form of a 30 MB csv file. But instead of using commas to separate fields, it used the pipe character (‘|’). I used DuckDB to quickly read the data from the file. I could have actually queried the file directly using DuckDB SQL, but in my case I saved it to a local DuckDB database and queried it from there.

Hope that helps.

Re: The DuckDB Local UI

#112

Post by setr »

drdaeman wrote: post #103
gnulinux wrote: post #48

Earlier quoted context omitted.

DuckDB is mind blowingly awesome. It is like SQLite, lightweight, embeddable, serverless, in-memory database, but it's optimized to be columnar (analytics optimized). It can work with files that are in filesystem, S3 etc without copying (it just looks at the necessary regions in the file) by just doing `select * from 's3://....something.parquet'`. It support automatic compression and automatic indexing. It can read j…

I'm sorry, I must be exceptionally stupid (or haven't seriously worked in this particular problem domain and thus lacking awareness), but I still can't figure out the use cases from this feature list. What sort of thing should I be working on, to think "oh, maybe I want this DuckDB thing here to do this for me?" I guess I don't really get the "that you want to learn something about" bit.

If you’re using SQLite already, then it’s the same use case but better at analytics

If you’re using excel power query and XLOOKUPs, then it’s similar but dramatically faster and without the excel autocorrection nonsense

If you’re doing data processing that fits on your local machine eg 50MB, 10GB, 50GB CSVs kind of thing, then it should be your default.

If you’re using pandas/numpy, this is probably better/faster/easier

Basically if you’re doing one-time data mangling tasks with quick python scripts or excel or similar, you should probably be looking at SQLite/duckdb.

For bigger/repeatable jobs, then just consider it a competitor to doing things with multiple CSV/JSON files.

Re: The DuckDB Local UI

#113

Post by daveguy »

plipt wrote: post #91
vamega wrote: post #25

This looks pretty great. The UI looked fantastic, and the post mentioned that it was open source. However what's open source appears to be the DuckDB extension, which forwards the requests to a remote URL. I've not been able to find the code for the actual UI. Is the actual UI open source, or is that something MotherDuck is allowing to be used by this while remaining proprietary? Right now it doesn't appear like this…

How is this promoted as a "local UI" if it gets the UI from a remote URL? Maybe the closed source UI is downloaded upon first execution for installation and then cached locally? Or is this a web app that loads from the remote URL each time?

It's a web interface, but it is served from the local machine. The default is http://localhost:4213/

See the note just above this link on data locations and the optional and explicit opt-in to motherduck:

https://duckdb.org/2025/03/12/duckdb-ui.html#features

Re: The DuckDB Local UI

#114

Post by tobilg »

markhalonen wrote: post #98
tobilg wrote: post #59

Earlier quoted context omitted.

Have a look at https://sql-workbench.com eventually, as it's using DuckDB WASM & Perspective to render the query results. Let me know what you think!

This is actually how I discovered Perspective!

Hahaha, nice. It's a small world.

Re: The DuckDB Local UI

#115

Post by liendolucas »

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and memory on my system run quite low (on a 8Gb system with a generous swap), sometimes even stopping the whole processing of the pipeline. For one heavy step we tried out DuckDB and I was blown away how performant against PySpark was. It was not only fast as hell but its memory footprint extremely low as well, almost as if something was wrong and had to recheck several times that it was correct, and yes it did what it was supposed to do. Now this is a place where I do actually care about how fast and performant a thing can be and not the nanoseconds that each JS frontend framework of the day claims to win. KUDOS to the DuckDB team.

Re: The DuckDB Local UI

#116

Post by Marazan »

liendolucas wrote: post #115

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and memory on my system run quite low (on a 8Gb system with a generous swap), sometimes even stopping the whole processing of the pipeline. For one heavy step we tried out DuckDB and I was blown away how performant against PySpark was. It was not only fast as hell b…

Spark is never going to be the right choice when running on a single system.

Spark is for when you have a hundreds of machines worth of processing to do

Re: The DuckDB Local UI

#117

Post by moooo99 »

Marazan wrote: post #116
liendolucas wrote: post #115

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and memory on my system run quite low (on a 8Gb system with a generous swap), sometimes even stopping the whole processing of the pipeline. For one heavy step we tried out DuckDB and I was blown away how performant against PySpark was. It was not only fast as hell b…

Spark is never going to be the right choice when running on a single system. Spark is for when you have a hundreds of machines worth of processing to do

> Spark is for when you have a hundreds of machines worth of processing to do

Absolutely agree. However, most uses of Spark I've seen in my career are people thinking they have hundreds of machines worth of processing to do.

Re: The DuckDB Local UI

#118

Post by Marazan »

moooo99 wrote: post #117
Marazan wrote: post #116

Earlier quoted context omitted.

Spark is never going to be the right choice when running on a single system. Spark is for when you have a hundreds of machines worth of processing to do

> Spark is for when you have a hundreds of machines worth of processing to do Absolutely agree. However, most uses of Spark I've seen in my career are people thinking they have hundreds of machines worth of processing to do.

And even when you jave quite a lot of machines worth of processing some single threaded streaming of data on a single machine can still beat out any distributed framework as the the overhead of distribution is large.

Re: The DuckDB Local UI

#119

Post by liendolucas »

Marazan wrote: post #116
liendolucas wrote: post #115

Anecdote. Last year I had to work with a heavy analytics process. The whole thing was 4 or 5 large steps and was written with PySpark. It was really slow and memory on my system run quite low (on a 8Gb system with a generous swap), sometimes even stopping the whole processing of the pipeline. For one heavy step we tried out DuckDB and I was blown away how performant against PySpark was. It was not only fast as hell b…

Spark is never going to be the right choice when running on a single system. Spark is for when you have a hundreds of machines worth of processing to do

We haven't developed the PySpark pipeline. It was given to us to be improved, which we did a whole rewrite to leave it more clean and understandable. We also tried a persistence switch to test if it was a better choice just in case a step failed we could resume from a prevoius one. I also had zero hands-on on PySpark and DuckDB. But yes, I was amazed at how far it was falling behind DuckDB. I wasn't expecting such a difference. Ah also this pipeline did indeed run on the cloud, but it was not posible to test it there, so the only choice was to run it locally.

Re: The DuckDB Local UI

#120

Post by davesque »

Weirdly, as cool as this looks, it's a bit concerning to me. It feels like this is marking a milestone in the history of a great open source project where they are doing one or many of the following:

1) Biting off more than they can chew,

2) Putting significant effort into something that's outside of their core value proposition,

3) Leaning more in the direction of supporting things with a for profit company that gradually cannibalizes the open source side.

Maybe I'm being too cynical. I hope I'm wrong.

Post reply on HN