Live data from Hacker News

Use DuckDB-WASM to query TB of data in browser

lil.law.harvard.edu

41–50 of 64 posts

Re: Use DuckDB-WASM to query TB of data in browser

#41
post #28

It's one of the best tricks in the book. We have been doing it for quite some time in our product to bring real time system observability with eBPF to the browser and have even found other techniques to really max-it-out beyond what you get off the shelf. https://yeet.cx

That's pretty cool. Any technical blog posts?

Re: Use DuckDB-WASM to query TB of data in browser

#43
post #28

It's one of the best tricks in the book. We have been doing it for quite some time in our product to bring real time system observability with eBPF to the browser and have even found other techniques to really max-it-out beyond what you get off the shelf. https://yeet.cx

That's pretty cool. Any technical blog posts?

we got a couple blog posts

https://yeet.cx/blog

Re: Use DuckDB-WASM to query TB of data in browser

#44
post #32
post #3

Earlier quoted context omitted.

S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.

Stick it behind Cloudflare and it should be effectively free.

Until it isn't.

Re: Use DuckDB-WASM to query TB of data in browser

#45
post #39

Earlier quoted context omitted.

> R2 is S3 compatible with no egress fees. There's no egress data transfer fees, but you still pay for the GET request operations. Lots of little range requests can add up quick.

Can't believe that is what the industry has come down to. Kind like clipping coupon to get the best deal according different pricing overlords. It is time like this that makes self-hosting a lot more attractive.

Luckily it's just static files. You can use whatever host you want.

Re: Use DuckDB-WASM to query TB of data in browser

#46
post #42

How… does it not blow up browser’s memory?

The UI element is a scrollable table with a fixed size viewport window, memory shouldn't be a problem since they just have to retrieve and cache a reasonable area around that window. Old data can just be discarded.

Re: Use DuckDB-WASM to query TB of data in browser

#48
post #2

OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

Can you replace S3 with a directory and nginx and save lot of money?

Re: Use DuckDB-WASM to query TB of data in browser

#49
post #2

OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

Can you replace S3 with a directory and nginx and save lot of money?

Yes, i.i.r.c. it's not S3 specific just URLs

Re: Use DuckDB-WASM to query TB of data in browser

#50
My company tried DuckDB-WASM + parquet + S3 a few months ago but we ended up stripping it all out and replacing it with a boring REST API.

On paper it seemed like a great fit, but it turned out the WASM build doesn't have feature-parity with the "normal" variant, so things that caused us to pick it like support for parquet compression and lazy loading were not supported. So it ended up not having great performance while introducing a lot of complexity, and also was terrible for first page load time due to needing the large WASM blob. Build pipeline complexity was also inherently higher due to the dependency and data packaging needed.

Just something to be aware of if you're thinking of using it. Our conclusion was that it wasn't worth it for most use cases, which is a shame because it seems like such a cool tech.

Post reply on HN