Use DuckDB-WASM to query TB of data in browser
lil.law.harvard.edu
Use DuckDB-WASM to query TB of data in browser
1–10 of 64 posts
Re: Use DuckDB-WASM to query TB of data in browser
#2OK, this is really neat:
- S3 is really cheap static storage for files.
- DuckDB is a database that uses S3 for its storage.
- WASM lets you run binary (non-JS) code in your browser.
- DuckDB-Wasm allows you to run a database in your browser.
Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
Re: Use DuckDB-WASM to query TB of data in browser
#3OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.
Re: Use DuckDB-WASM to query TB of data in browser
#4OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
Or use R2 instead. It’s even easier.
Re: Use DuckDB-WASM to query TB of data in browser
#5OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
S3 is doing quite a lot of sophisticated lifting to qualify as no backend at all.
But yeah - this is pretty neat. Easily seems like the future of static datasets should wind up in something like this. Just data, with some well chosen indices.
Re: Use DuckDB-WASM to query TB of data in browser
#6Neat. Can you use duckdb backed on another store like rocksdb or something? Also, I wonder how one stops ddos. Put the whole thing behind Cloudflare?
Re: Use DuckDB-WASM to query TB of data in browser
#7I tried DuckDB - liked it a lot - was ready to go further.
But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use.
This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors.
And then there was some indexed bug that crashed seemingly unrelated to memory.
Life is too short for crashy database software so I reluctantly dropped it. I was disappointed because it was exactly what I was looking for.
Re: Use DuckDB-WASM to query TB of data in browser
#8OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
S3 is doing quite a lot of sophisticated lifting to qualify as no backend at all. But yeah - this is pretty neat. Easily seems like the future of static datasets should wind up in something like this. Just data, with some well chosen indices.
Still qualifies imo. Everything is static and on a CDN.
Lack of server/dynamic code qualifies as no backend.
Re: Use DuckDB-WASM to query TB of data in browser
#9OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.
R2 is S3 compatible with no egress fees.
Cloudflare actually has built in iceberg support for R2 buckets. It's quite nice.
Combine that with their pipelines it's a simple http request to ingest, then just point duckdb to the iceberg enabled R2 bucket to analyze.
Re: Use DuckDB-WASM to query TB of data in browser
#10I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
+1 this was my experience trying it out as well. I find that for getting started and for simple usecases it works amazing. But I have quite a lot of concerns about how it scales to more complex and esoteric workloads.
Non-deterministic OOMs especially are some of the worst things in the sort of tools I'd want to use DuckDB in and as you say, I found it to be more common than I would like.