Understanding every byte in a WASM module • Hacker News Board

Re: Understanding every byte in a WASM module

#31

Post by kevingadd » Sat, Dec 23, 2023, 10:23 PM UTC

amelius wrote: post #4

rockwotj wrote: post #2

This is pretty great! I highly recommend building a parser for Wasm, it shows off the nice design that it can be compiled in a streaming fashion and other nice properties. I built a small parser for a wasm runtime I was going to hack at, although that lost steam and now I just contribute to wasmtime, but it was a great way to learn. The source if you’re interested in C++20 and coroutines: https://github.com/rockwotj/…

> the nice design But integers are read 7 bits at a time. Wouldn't it be much easier to use 8 bits?

It would be. The use of varints in wasm complicates encoding and decoding and introduces ambiguity into the format for cases like v128 constants (which despite technically being 128-bit integers, are not embedded using leb128.) It also makes it harder to write a single-pass compiler.

It also isn't much of a wire size improvement, since the overall format design assumes you will compress it using brotli (or gzip, I guess) - there are lots of opportunities to make the format smaller or arrange it in a way that makes it more compressible, and the decision was made not to do those things since brotli is really good at compressing data.

I think the people who took control of the binary format just really liked varints for some reason, so we ended up stuck with them. It's at least not particularly harmful, it's just a small waste of everyone's time. Thankfully, my understanding is that in practice most modules will only get decoded once or twice before the compiled code is cached by your browser.

Re: Understanding every byte in a WASM module

#32

Post by BonoboIO » Sun, Dec 24, 2023, 2:47 AM UTC

Hackbraten wrote: post #24

flohofwoe wrote: post #17

Earlier quoted context omitted.

The browser devtools can give you a disassembled view of the WebAssembly byte code (basically the wasm2wat output) and also allow to set breakpoints and debug-step through the disassembly. Since WASM is a "structured-programming ISA" with code blocks that can be intended, it's actually quite convenient.

Thanks for the pointer! The WASM payload itself appears to be obfuscated. The player seems to deobfuscate it at runtime, dynamically load it as a WASM module, execute it and then destroy it again. I'm having a hard time pinpointing the exact moment where the WASM is being loaded (because the JS that performs all those things is horribly obfuscated, too.) The disassembled view in the dev tools might come in handy to t…

Sky?

Re: Understanding every byte in a WASM module

#33

Post by WaffleIronMaker » Sun, Dec 24, 2023, 4:08 AM UTC

marianoguerra wrote: post #29

If you are interested in this type of approach you may like https://wasmgroundup.com/ > This book takes a hands-on, bottoms-up approach: you'll go from hand crafting bytecodes to writing a real compiler for a simple programming language. disclaimer: I'm the co-author

I'm impressed with the pricing of $19 early access / $39 final version.

I'm used to high-value books for professionals either being completely free,[1] or starting at $100. [2][3] It seems like a good compromise at that price.

[1] https://craftinginterpreters.com/contents.html

[2] https://www.refactoringui.com/#get-refactoring-ui

[3] https://www.amazon.com/dp/0262048434

Re: Understanding every byte in a WASM module

#34

Post by Hackbraten » Sun, Dec 24, 2023, 10:16 AM UTC

BonoboIO wrote: post #32

Hackbraten wrote: post #24

Earlier quoted context omitted.

Thanks for the pointer! The WASM payload itself appears to be obfuscated. The player seems to deobfuscate it at runtime, dynamically load it as a WASM module, execute it and then destroy it again. I'm having a hard time pinpointing the exact moment where the WASM is being loaded (because the JS that performs all those things is horribly obfuscated, too.) The disassembled view in the dev tools might come in handy to t…

Sky?

NowTV (aka WowTV in German-speaking countries). Part of the code mentions Sky.

Re: Understanding every byte in a WASM module

#35

Post by p1mrx » Sun, Dec 24, 2023, 11:42 PM UTC

joquarky wrote: post #28

teraflop wrote: post #6

Earlier quoted context omitted.

If you use all 8 bits of a byte for data, you need a separate way to store the length, or you need to make everything fixed-width which wastes space. "Varints" or variable-length integers, with 7 bits of data and 1 continuation bit per byte, are used in a fairly wide variety of formats (e.g. Protobuf, SQLite and Lucene). They're compact when most of your integers are small, while still being quite simple and fast to…

I wonder if something resembling this could have been better than how IPv6 attempts to solve the address space problem.

Variable-length IP addresses were considered at the time: https://datatracker.ietf.org/doc/html/rfc1752#section-10.2

Re: Understanding every byte in a WASM module

#36

Post by circuit10 » Mon, Dec 25, 2023, 6:50 AM UTC

lioeters wrote: post #13

danielvaughn wrote: post #11

Earlier quoted context omitted.

Very cool. You might be interested in AssemblyScript, if you haven't heard of them before. They did something very similar: https://www.assemblyscript.org/

Ah yes, I'd love to get more familiar with AssemblyScript too. One thing I like about the afore-mentioned WAT compiler though, is that it's simple to run in the browser, which the AssemblyScript compiler doesn't support, if I understand correctly. That characteristic of the compiler would be useful if I want to build a higher-level language on top of it, for example with a web playground (code editor and runtime).

The AssemblyScript compiler runs in a browser too, that’s how the demo on the homepage works

Re: Understanding every byte in a WASM module

#37

Post by lioeters » Mon, Dec 25, 2023, 6:41 PM UTC

circuit10 wrote: post #36

lioeters wrote: post #13

Earlier quoted context omitted.

Ah yes, I'd love to get more familiar with AssemblyScript too. One thing I like about the afore-mentioned WAT compiler though, is that it's simple to run in the browser, which the AssemblyScript compiler doesn't support, if I understand correctly. That characteristic of the compiler would be useful if I want to build a higher-level language on top of it, for example with a web playground (code editor and runtime).

The AssemblyScript compiler runs in a browser too, that’s how the demo on the homepage works

Thanks for mentioning it, I dug into the website of AssemblyScript and found that the compiler does indeed run in the browser. The demo loads a web-build like:

  
  
  
  import asc from "assemblyscript/asc";

And how they compile AssemblyScript to WAT/WASM/JS:

https://github.com/AssemblyScript/website/blob/main/src/.vue...

Here's how the compiler is built to target the web.

https://github.com/AssemblyScript/assemblyscript/blob/main/s...

How fun! This inspires me to explore further what I could build with it.

Re: Understanding every byte in a WASM module

#38

Post by pcmoore » Tue, Dec 26, 2023, 2:12 PM UTC

I entered into WASM assuming it was simple and minimal but there are a lot of runtime edge cases that are quite surprising. I built an interpreter that passes all the non-SIMD test cases from the official spec. Would be interesting to clean-up the spec to be a real minimal execution only core. There is a real barrier to entry for producing new lightweight implementations.

https://github.com/peterseymour/winter