Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use DuckDB-WASM to query TB of data in browser

lil.law.harvard.edu

Post reply on HN
Print view View original

31–40 of 64 posts

Re: Use DuckDB-WASM to query TB of data in browser

#31

Post by apwheele »

theultdev wrote: post #9
timeflex wrote: post #3

Earlier quoted context omitted.

S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.

R2 is S3 compatible with no egress fees. Cloudflare actually has built in iceberg support for R2 buckets. It's quite nice. Combine that with their pipelines it's a simple http request to ingest, then just point duckdb to the iceberg enabled R2 bucket to analyze.

For a demo of this (although not sure with duckdb wasm that it works with iceberg) https://andrewpwheeler.com/2025/06/29/using-duckdb-wasm-clou...

Re: Use DuckDB-WASM to query TB of data in browser

#32

Post by simonw »

timeflex wrote: post #3
mlissner wrote: post #2

OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.

Stick it behind Cloudflare and it should be effectively free.

Re: Use DuckDB-WASM to query TB of data in browser

#33

Post by simonw »

thadt wrote: post #5
mlissner wrote: post #2

OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

S3 is doing quite a lot of sophisticated lifting to qualify as no backend at all. But yeah - this is pretty neat. Easily seems like the future of static datasets should wind up in something like this. Just data, with some well chosen indices.

I believe all S3 has to do here is respond to HTTP Range queries, which are supported by almost every static server out there - Apache, Nginx etc should all support the same trick.

Re: Use DuckDB-WASM to query TB of data in browser

#34

Post by simonw »

SteveMoody73 wrote: post #16

My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools

What appropriate tool would you use for this instead?

Re: Use DuckDB-WASM to query TB of data in browser

#35

Post by whalesalad »

leetrout wrote: post #29

I built something on top of DuckDB last year but it never got deployed. They wanted to trust Postgres. I didn't use the in browser WASM but I did expose an api endpoint that passed data exploration queries directly to the backend like a knock off of what new relic does. I also use that same endpoint for all the graphs and metrics in the UI. DuckDB is phenomenal tech and I love to use it with data ponds instead of dat…

Cool thing about DuckDB is it can be embedded. We have a data pipeline that produces a duckdb file and puts it on S3. The app periodically checks that assets etag and pulls it down when it changes. Most of our DB interactions use PSQL, but we have one module that leverages DuckDB and this file for reads. So it's definitely not all-or-nothing.

Re: Use DuckDB-WASM to query TB of data in browser

#36

Post by shawn-butler »

SteveMoody73 wrote: post #16

My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools

I doubt they are querying 1 TB of data in the browser. DuckDB-WASM issues http range requests on behalf of client to request only the bytes required, especially handy with parquet files (columnar format) that will exclude columns you don't even need.

But the article is a little light on technical details. In some cases it might make sense to bring the entire file client-side.

Re: Use DuckDB-WASM to query TB of data in browser

#37

Post by thadt »

simonw wrote: post #33
thadt wrote: post #5

Earlier quoted context omitted.

S3 is doing quite a lot of sophisticated lifting to qualify as no backend at all. But yeah - this is pretty neat. Easily seems like the future of static datasets should wind up in something like this. Just data, with some well chosen indices.

I believe all S3 has to do here is respond to HTTP Range queries, which are supported by almost every static server out there - Apache, Nginx etc should all support the same trick.

100%. I’m with y’all - this is what I would also call a “no-backend” solution and I’m all in on this type of approach for static data sets - this is the future, and could be served with a very simple web server.

I’m just bemused that we all refer to one of the larger, more sophisticated storage systems on the plant, composed of dozens of subsystems and thousands of servers as “no backend at all.” Kind of a “draw the rest of the owl”.

Re: Use DuckDB-WASM to query TB of data in browser

#38

Post by 8organicbits »

theultdev wrote: post #9
timeflex wrote: post #3

Earlier quoted context omitted.

S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.

R2 is S3 compatible with no egress fees. Cloudflare actually has built in iceberg support for R2 buckets. It's quite nice. Combine that with their pipelines it's a simple http request to ingest, then just point duckdb to the iceberg enabled R2 bucket to analyze.

> R2 is S3 compatible with no egress fees.

There's no egress data transfer fees, but you still pay for the GET request operations. Lots of little range requests can add up quick.

Re: Use DuckDB-WASM to query TB of data in browser

#39

Post by zenmac »

8organicbits wrote: post #38
theultdev wrote: post #9

Earlier quoted context omitted.

R2 is S3 compatible with no egress fees. Cloudflare actually has built in iceberg support for R2 buckets. It's quite nice. Combine that with their pipelines it's a simple http request to ingest, then just point duckdb to the iceberg enabled R2 bucket to analyze.

> R2 is S3 compatible with no egress fees. There's no egress data transfer fees, but you still pay for the GET request operations. Lots of little range requests can add up quick.

Can't believe that is what the industry has come down to. Kind like clipping coupon to get the best deal according different pricing overlords.

It is time like this that makes self-hosting a lot more attractive.

Re: Use DuckDB-WASM to query TB of data in browser

#40

Post by zenmac »

whalesalad wrote: post #35
leetrout wrote: post #29

I built something on top of DuckDB last year but it never got deployed. They wanted to trust Postgres. I didn't use the in browser WASM but I did expose an api endpoint that passed data exploration queries directly to the backend like a knock off of what new relic does. I also use that same endpoint for all the graphs and metrics in the UI. DuckDB is phenomenal tech and I love to use it with data ponds instead of dat…

Cool thing about DuckDB is it can be embedded. We have a data pipeline that produces a duckdb file and puts it on S3. The app periodically checks that assets etag and pulls it down when it changes. Most of our DB interactions use PSQL, but we have one module that leverages DuckDB and this file for reads. So it's definitely not all-or-nothing.

Are you using pg_duckdb to embedded it inside postgres and access it via psql or other pg clients?
Post reply on HN