Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding every byte in a WASM module

danielmangum.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: Understanding every byte in a WASM module

#21

Post by xjay »

amelius wrote: post #4
rockwotj wrote: post #2

This is pretty great! I highly recommend building a parser for Wasm, it shows off the nice design that it can be compiled in a streaming fashion and other nice properties. I built a small parser for a wasm runtime I was going to hack at, although that lost steam and now I just contribute to wasmtime, but it was a great way to learn. The source if you’re interested in C++20 and coroutines: https://github.com/rockwotj/…

> the nice design But integers are read 7 bits at a time. Wouldn't it be much easier to use 8 bits?

In this case, a 64-bit little-endian unsigned integer Ought to be enough for anyone. [1]

Using variable-length "micro-compression" schemes like these merges concerns [2], add processing overhead, and lose out on the simplicity and efficiency of working on the data directly in memory, maybe even reusing the same memory/cache line after checking identifiers, etc.

The 7-bit VLQ scheme [3] makes sense for MIDI sequences (1980s), as they were stored on MIDI hardware with limited memory, and had basic processing needs.

[1] https://quoteinvestigator.com/2011/09/08/640k-enough/

[2] https://en.wikipedia.org/wiki/Separation_of_concerns

[3] https://en.wikipedia.org/wiki/Variable-length_quantity

Re: Understanding every byte in a WASM module

#22

Post by captn3m0 »

evmar wrote: post #12

If you're looking into a .wasm file I wrote an interactive viewer called weave[1]. Here is a demo with some Rust output: https://evmar.github.io/weave/?wasm/rust.wasm be sure to click on 'code' to dive into individual functions. [1] https://github.com/evmar/weave

There’s also https://wasdk.github.io/wasmcodeexplorer/ which highlights the bytes.

Re: Understanding every byte in a WASM module

#23

Post by monocasa »

xjay wrote: post #21
amelius wrote: post #4

Earlier quoted context omitted.

> the nice design But integers are read 7 bits at a time. Wouldn't it be much easier to use 8 bits?

In this case, a 64-bit little-endian unsigned integer Ought to be enough for anyone. [1] Using variable-length "micro-compression" schemes like these merges concerns [2], add processing overhead, and lose out on the simplicity and efficiency of working on the data directly in memory, maybe even reusing the same memory/cache line after checking identifiers, etc. The 7-bit VLQ scheme [3] makes sense for MIDI sequences…

Eh, I don't think the .wasm file is intended to be an in memory format. For an on wire/storage format, variable length integers like protobuf uses is a perfectly valid choice.

Re: Understanding every byte in a WASM module

#24

Post by Hackbraten »

flohofwoe wrote: post #17
Hackbraten wrote: post #5

I really appreciate this article. I’ve noticed that modern streaming web players have started using WASM as part of their obfuscation toolbox. Understanding WASM modules is important for reverse engineering.

The browser devtools can give you a disassembled view of the WebAssembly byte code (basically the wasm2wat output) and also allow to set breakpoints and debug-step through the disassembly. Since WASM is a "structured-programming ISA" with code blocks that can be intended, it's actually quite convenient.

Thanks for the pointer!

The WASM payload itself appears to be obfuscated. The player seems to deobfuscate it at runtime, dynamically load it as a WASM module, execute it and then destroy it again. I'm having a hard time pinpointing the exact moment where the WASM is being loaded (because the JS that performs all those things is horribly obfuscated, too.)

The disassembled view in the dev tools might come in handy to tackle this. My goal is to obtain their secret HMAC key so I can change "Linux" into whatever the server prefers to hear, and re-sign the HMAC for that manipulated HTTP request before the player sends it.

(All that effort so I finally get to watch my Bundesliga streams again, for which I'm paying them 30 EUR a month.)

Re: Understanding every byte in a WASM module

#25

Post by Mindless2112 »

flohofwoe wrote: post #19
svat wrote: post #7

Earlier quoted context omitted.

This is basically the design of UTF-8 or varint encoding. The integer's representation is read one byte (8 bits) at a time, which can then be interpreted by looking at the first bit which says whether the integer is "done" or more bytes need to be read. (If we were using all 8 bits for the representation and an additional leading bit for whether it continues, we'd end up having to read 9 bits or something, which is a…

UTF-8 is a bit more complex than that. The first byte in a sequence basically tells you how many bytes are following by looking at the top-most bits. 0xxxxxxx => no followup bytes, just the 7-bit ASCII code 110xxxxx => 1 byte follows 1110xxxx => 2 bytes follow 11110xxx => 3 bytes follow The followup bytes then always have the pattern 10xxxxxx. E.g. the only bytes that have bit 7 cleared are the single-byte 7-bit ASCI…

A UTF-8-like encoding would probably be better than VLQ encoding because the length is known up-front, which allows for the decoder to have fewer branches.

Something like [1] or alternatively [2]. See also [3].

[1] https://en.wikipedia.org/wiki/UTF-8#FSS-UTF [2] https://sqlite.org/src4/doc/trunk/www/varint.wiki [3] https://news.ycombinator.com/item?id=11263378

Re: Understanding every byte in a WASM module

#26

Post by hoten »

Hackbraten wrote: post #24
flohofwoe wrote: post #17

Earlier quoted context omitted.

The browser devtools can give you a disassembled view of the WebAssembly byte code (basically the wasm2wat output) and also allow to set breakpoints and debug-step through the disassembly. Since WASM is a "structured-programming ISA" with code blocks that can be intended, it's actually quite convenient.

Thanks for the pointer! The WASM payload itself appears to be obfuscated. The player seems to deobfuscate it at runtime, dynamically load it as a WASM module, execute it and then destroy it again. I'm having a hard time pinpointing the exact moment where the WASM is being loaded (because the JS that performs all those things is horribly obfuscated, too.) The disassembled view in the dev tools might come in handy to t…

Ctrl+f for the WASM compile function (or maybe replace it with a wrapper function that has a debugger statement), breakpoint there, step over, then look in the Sources panel. You'll find the WAT for your secret wasm then.

It may even still be in Sources panel post-deletion, in which case no need for any of that.

Re: Understanding every byte in a WASM module

#27

Post by dataangel »

monocasa wrote: post #23
xjay wrote: post #21

Earlier quoted context omitted.

In this case, a 64-bit little-endian unsigned integer Ought to be enough for anyone. [1] Using variable-length "micro-compression" schemes like these merges concerns [2], add processing overhead, and lose out on the simplicity and efficiency of working on the data directly in memory, maybe even reusing the same memory/cache line after checking identifiers, etc. The 7-bit VLQ scheme [3] makes sense for MIDI sequences…

Eh, I don't think the .wasm file is intended to be an in memory format. For an on wire/storage format, variable length integers like protobuf uses is a perfectly valid choice.

It's actually the protobuf design mistake that most adversely affects decoding performance. The original creator who went on to make Cap'N Proto makes the excellent point that it's better to leave the extra bytes and then just slap a compressor that only works on runs of 0 bytes on top. Faster for the same savings. There are also newer schemes from Daniel Lemire and others more amenable to SIMD decoding.

Re: Understanding every byte in a WASM module

#28

Post by joquarky »

teraflop wrote: post #6
amelius wrote: post #4

Earlier quoted context omitted.

> the nice design But integers are read 7 bits at a time. Wouldn't it be much easier to use 8 bits?

If you use all 8 bits of a byte for data, you need a separate way to store the length, or you need to make everything fixed-width which wastes space. "Varints" or variable-length integers, with 7 bits of data and 1 continuation bit per byte, are used in a fairly wide variety of formats (e.g. Protobuf, SQLite and Lucene). They're compact when most of your integers are small, while still being quite simple and fast to…

I wonder if something resembling this could have been better than how IPv6 attempts to solve the address space problem.

Re: Understanding every byte in a WASM module

#29

Post by marianoguerra »

If you are interested in this type of approach you may like https://wasmgroundup.com/

> This book takes a hands-on, bottoms-up approach: you'll go from hand crafting bytecodes to writing a real compiler for a simple programming language.

disclaimer: I'm the co-author

Re: Understanding every byte in a WASM module

#30

Post by monocasa »

dataangel wrote: post #27
monocasa wrote: post #23

Earlier quoted context omitted.

Eh, I don't think the .wasm file is intended to be an in memory format. For an on wire/storage format, variable length integers like protobuf uses is a perfectly valid choice.

It's actually the protobuf design mistake that most adversely affects decoding performance. The original creator who went on to make Cap'N Proto makes the excellent point that it's better to leave the extra bytes and then just slap a compressor that only works on runs of 0 bytes on top. Faster for the same savings. There are also newer schemes from Daniel Lemire and others more amenable to SIMD decoding.

Sure, but that decoding speed penalty doesn't stop protobuf (and similar encodings using leb128 like thrift) from being some of the most used binary encodings on the planet, even in relatively low latency spaces.

Additionally, Cap'N Proto RLE encodes the bytestream, which is pretty similar to the overhead of LEB128 integers.

And on top of all of that, .wasm isn't intended to be an IPC format, so the fact that it's choices are typical of that space (despite not being leading edge) reflect very well for the space it's intended to be in.

Post reply on HN