Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

motifanalytics.medium.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#31

Post by pmm »

Author here. Thank you all for the comments. I take full responsibility for stupidly using an image for posting the code snippet. Sorry for that! Also, the article was originally posted almost 2 years ago (and "resurrected" with the recent migration to Medium). This is why a fairly old DuckDB version is referenced there. Some of the issues I observed are now gone too.

Obviously, many things have changed since then. We've experimented extensively and moved back and forth with using DuckDB for our internal cloud processing architecture. We eventually settled on just using it for reading the data and then handling everything else in custom workers. Even using TypeScript, we achieved close to 1M events/s per worker overall with very high scalability. However, our use-case is quite distinct. We use a custom query engine (for sequence processing), which has driven many design decisions.

Overall, I think DuckDB (both vanilla and WASM version) is absolutely phenomenal. It also matured since my original blog post. I believe we'll only see more and more projects using it as their backbone. For example, MotherDuck is doing some amazing things with it (e.g., https://duckdb.org/2023/03/12/duckdb-ui) but there are also many more exciting initiatives.

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#32

Post by jillyboel »

jillyboel wrote: post #20
jasmcole wrote: post #17

We use the WASM build of DuckDB quite extensively at Count ( https://count.co - 2-3m queries per month). There are a couple of bugs we've noticed, but given that it's pretty much maintained by a single person seems impressively reliable!

Looking at your insane pricing page I have to assume that you are sponsoring that single person?

Gotta love being downvoted for daring to suggest a company should sponsor the sole open source dev making their whole product possible.

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#33

Post by fragmede »

mcraiha wrote: post #5

Tip for all the blog authors, do NOT post code as image. Specially do not add fake editor UI and drop shadow to the image. In this case 25 lines of code is 50 kB of image binary. Also it cannot be searched via search engine. Nor can it be read with screen reader.

OCR's come pretty far these days and I can select text off an image with my iPhone processing it locally with a fair bit of success.

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#34

Post by drtgh »

mcraiha wrote: post #5

Tip for all the blog authors, do NOT post code as image. Specially do not add fake editor UI and drop shadow to the image. In this case 25 lines of code is 50 kB of image binary. Also it cannot be searched via search engine. Nor can it be read with screen reader.

As long as such 50kB image doesn’t get replaced with 700kB of javascript bundles for coloring code.

I mean, if such coloring it's going to be done, it should be done with HTML/CSS.

For the OP article, for screenreaders perhaps you are sugesting people to use the alt attribute or similar.

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#35

Post by azakai »

geokon wrote: post #30
azakai wrote: post #27

> [wasm] is executed in a stack-based virtual machine rather than as a native library code. Wasm's binary format is indeed a stack-based virtual machine, but that is not how it is executed. Optimizing VMs convert it to SSA form, basic blocks, and finally machine code, much the same as clang or gcc compile native library code. It is true that wasm has some overhead, but that is due to portability and sandboxing, not t…

The more I read about WASM the more it sounds like the JVM I'm still not clear what at its core it's done differently (in a way that couldn't be bolted on to a subset of the JVM)

The JVM is designed around Java. That's really the main difference, and it brings some downsides for the goals of wasm, which include running native code - think C++ or Rust. The JVM is great at Java, which relies on runtime inlining etc., but not at C++, which assumes ahead-of-time inlining and other optimizations.

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#36

Post by geokon »

azakai wrote: post #35
geokon wrote: post #30

Earlier quoted context omitted.

The more I read about WASM the more it sounds like the JVM I'm still not clear what at its core it's done differently (in a way that couldn't be bolted on to a subset of the JVM)

The JVM is designed around Java. That's really the main difference, and it brings some downsides for the goals of wasm, which include running native code - think C++ or Rust. The JVM is great at Java, which relies on runtime inlining etc., but not at C++, which assumes ahead-of-time inlining and other optimizations.

I don't understand how the virtual machine would preclude you from in-lining ahead of time...? That's done when you're compiling.

What is WASM doing to facilitate recompiling native code that isn't practical to do on the JVM

Re: My Browser WASM't Prepared for This. Using DuckDB, Apache Arrow and Web Workers

#37

Post by ralferoo »

brulard wrote: post #26
shepmaster wrote: post #9

Earlier quoted context omitted.

We made this to be used as a reply when pictures are misused where text would be better: https://fewer.pics/

It should have been animated.

It should have been a .swf and required a plugin to be installed to see it.
Post reply on HN