Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use DuckDB-WASM to query TB of data in browser

lil.law.harvard.edu

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Use DuckDB-WASM to query TB of data in browser

#51

Post by fragmede »

shawn-butler wrote: post #36
SteveMoody73 wrote: post #16

My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools

I doubt they are querying 1 TB of data in the browser. DuckDB-WASM issues http range requests on behalf of client to request only the bytes required, especially handy with parquet files (columnar format) that will exclude columns you don't even need. But the article is a little light on technical details. In some cases it might make sense to bring the entire file client-side.

For small databases, SQLite is handy, as there are multiple ways to parse the format for clients.

Re: Use DuckDB-WASM to query TB of data in browser

#52

Post by mentalgear »

dtech wrote: post #50

My company tried DuckDB-WASM + parquet + S3 a few months ago but we ended up stripping it all out and replacing it with a boring REST API. On paper it seemed like a great fit, but it turned out the WASM build doesn't have feature-parity with the "normal" variant, so things that caused us to pick it like support for parquet compression and lazy loading were not supported. So it ended up not having great performance wh…

> WASM build doesn't have feature-parity with the "normal" variant

It's a good point, but the wasm docs state that feature-parity isn't there - yet. It could certainly be more detailed, but it seems strange that your company would do all this work without first checking the feature-coverage / specs.

> WebAssembly is basically an additional platform, and there might be platform-specific limitations that make some extensions not able to match their native capabilities or to perform them in a different way.

https://duckdb.org/docs/stable/clients/wasm/extensions

Re: Use DuckDB-WASM to query TB of data in browser

#53

Post by dtech »

mentalgear wrote: post #52
dtech wrote: post #50

My company tried DuckDB-WASM + parquet + S3 a few months ago but we ended up stripping it all out and replacing it with a boring REST API. On paper it seemed like a great fit, but it turned out the WASM build doesn't have feature-parity with the "normal" variant, so things that caused us to pick it like support for parquet compression and lazy loading were not supported. So it ended up not having great performance wh…

> WASM build doesn't have feature-parity with the "normal" variant It's a good point, but the wasm docs state that feature-parity isn't there - yet. It could certainly be more detailed, but it seems strange that your company would do all this work without first checking the feature-coverage / specs. > WebAssembly is basically an additional platform, and there might be platform-specific limitations that make some exte…

Note that your docs specifically mentions parquet was supported, but we found out the hard way some specific features turned out not to be supported with WASM + parquet. I did a quick glance at your docs and could not find references to that, so I'm not surprised it was missed.

It was a project that exploited a new opportunity so time-to-market was the most important thing, I'm not suprised these things were missed, and replacing the data loading mechanism was maybe 1 week of work for 1 person, so it wasn't that impactful a change later.

Re: Use DuckDB-WASM to query TB of data in browser

#54

Post by mpweiher »

codedokode wrote: post #48
mlissner wrote: post #2

OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.

Can you replace S3 with a directory and nginx and save lot of money?

Yes. Especially if you use Storage Combinators.

They let you easily abstract over storage.

https://2019.splashcon.org/details/splash-2019-Onward-papers...

Re: Use DuckDB-WASM to query TB of data in browser

#55

Post by ludicrousdispla »

dtech wrote: post #50

My company tried DuckDB-WASM + parquet + S3 a few months ago but we ended up stripping it all out and replacing it with a boring REST API. On paper it seemed like a great fit, but it turned out the WASM build doesn't have feature-parity with the "normal" variant, so things that caused us to pick it like support for parquet compression and lazy loading were not supported. So it ended up not having great performance wh…

DuckDB-WASM supports parquet file decompression though, so if you have a backend process generating them it's a non issue.

How large was your WASM build? I'm using the standard duckdb-wasm, along with JS functions to form the SQL queries, and not seeing onerous load times.

Re: Use DuckDB-WASM to query TB of data in browser

#56

Post by mentalgear »

dtech wrote: post #53
mentalgear wrote: post #52

Earlier quoted context omitted.

> WASM build doesn't have feature-parity with the "normal" variant It's a good point, but the wasm docs state that feature-parity isn't there - yet. It could certainly be more detailed, but it seems strange that your company would do all this work without first checking the feature-coverage / specs. > WebAssembly is basically an additional platform, and there might be platform-specific limitations that make some exte…

Note that your docs specifically mentions parquet was supported, but we found out the hard way some specific features turned out not to be supported with WASM + parquet. I did a quick glance at your docs and could not find references to that, so I'm not surprised it was missed. It was a project that exploited a new opportunity so time-to-market was the most important thing, I'm not suprised these things were missed,…

Fair point, thx for sharing your experiences ! You might want to edit the duck-wasm docs in that regard to alert others/the team of this constraint.

Re: Use DuckDB-WASM to query TB of data in browser

#57

Post by ngc6677 »

Also similar procedure used on joblist.today https://github.com/joblisttoday to fetch hiring companies and their jobs and store them into sqlite and duckdb, and retrieved on the client side with their wasm modules. The database are generated with a daily github workflow and hosted as artifact on a github page.

Re: Use DuckDB-WASM to query TB of data in browser

#58

Post by victor106 »

leetrout wrote: post #29

I built something on top of DuckDB last year but it never got deployed. They wanted to trust Postgres. I didn't use the in browser WASM but I did expose an api endpoint that passed data exploration queries directly to the backend like a knock off of what new relic does. I also use that same endpoint for all the graphs and metrics in the UI. DuckDB is phenomenal tech and I love to use it with data ponds instead of dat…

> data ponds instead of data lakes

What are data ponds? Never heard the term before

Re: Use DuckDB-WASM to query TB of data in browser

#59

Post by vikramkr »

barrenko wrote: post #47

Where do I learn how to set up this sort of stuff? Trial and error? I kinda never need it for personal projects (so far), which always leads me to forget this stuff in between jobs kinda quickly. Is there a decent book?

If you want to learn it the best way is probably to come up with a personal project idea that requires it specifically? Idk how much you'd get out of a book but you could always do a side project with the specific goal of doing it just to learn a particular stack or whatever

Re: Use DuckDB-WASM to query TB of data in browser

#60

Post by leetrout »

victor106 wrote: post #58
leetrout wrote: post #29

I built something on top of DuckDB last year but it never got deployed. They wanted to trust Postgres. I didn't use the in browser WASM but I did expose an api endpoint that passed data exploration queries directly to the backend like a knock off of what new relic does. I also use that same endpoint for all the graphs and metrics in the UI. DuckDB is phenomenal tech and I love to use it with data ponds instead of dat…

> data ponds instead of data lakes What are data ponds? Never heard the term before

Haha, my term. Somewhere between a data lake and warehouse - still unstructured but not _everything_ in one place. For instance, if I have a multi-tenant app I might choose to have a duckdb setup for each customer with pre-filtered data living alongside some global unstructured data.

Maybe there's already a term that covers this but I like the imagery of the metaphor... "smaller, multiple data but same idea as the big one".

Post reply on HN