Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Weird things I learned while writing an x86 emulator

timdbg.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Weird things I learned while writing an x86 emulator

#41

Post by ginnungagap »

wruza wrote: post #35
kuroguro wrote: post #34

Earlier quoted context omitted.

Used the exact same trick when researching dumb ways to break AV signatures at uni :) You can also abuse the displacement math with eiz: ff 34 24 push DWORD PTR [esp] ff 34 e4 push DWORD PTR [esp+eiz*8] Or some useless prefixes may work: 80 c0 53 add al,0x53 36 04 53 ss add al,0x53

This is the first time I learned about eiz. Cool trick! What’s eiz: https://stackoverflow.com/a/2553556/3125367

If I remember correctly there are also tricks like

  lea rax, [eiz+rbx*4]
instead of

  lea rax, [rbx*4]
because the latter will produce an heavier binary by being interpreted as

  lea rax, [00000000h+rbx*4]

Re: Weird things I learned while writing an x86 emulator

#42

Post by Liquid_Fire »

menaerus wrote: post #22

Has anyone used TTD, or rr for that purpose in Linux world, to debug a complex multi-threaded application? What is the main use-case of these tools? For example, limitations of rr seem to suggest that it is almost of no use for multi-threaded programs so I have never actually tried it. I don't know about the TTD though. > rr limitations > ... > emulates a single-core machine. So, parallel programs incur the slowdown…

> limitations of rr seem to suggest that it is almost of no use for multi-threaded programs

Only if your main use case is debugging race conditions only possible with multiple cores, which is a tiny subset of all bugs.

I use rr all the time for a complex multi-threaded (although not extremely parallel) application and it works wonders. It frequently saves me hours of debugging. Practically none of the issues I use it for are race conditions (not because it doesn't work well for those, but because I rarely get any).

Even if I had to suffer an 8x slowdown from running it on a single core, it would still be worth it nearly every time.

Re: Weird things I learned while writing an x86 emulator

#43

Post by m3kw9 »

woodruffw wrote: post #7

This is a good list! Another fun quirk: because x86 is a register-memory architecture and allows all kinds of variants of reg/mem operand encodings, there are a handful of equivalent encodings with exactly the same lengths (and just slightly different ModR/M bytes). You can take advantage of this to do software fingerprinting or, in my case, steganography without changing an executable’s size or semantics[1]. [1]: ht…

Why would I want to hide a message beside a program?

Re: Weird things I learned while writing an x86 emulator

#44

Post by praptak »

woodruffw wrote: post #7

This is a good list! Another fun quirk: because x86 is a register-memory architecture and allows all kinds of variants of reg/mem operand encodings, there are a handful of equivalent encodings with exactly the same lengths (and just slightly different ModR/M bytes). You can take advantage of this to do software fingerprinting or, in my case, steganography without changing an executable’s size or semantics[1]. [1]: ht…

The article itself exposes some encoding redundancy too: "[...] the count is masked against 1FH, essentially using only the lowest five bits".

So each rotation instruction is 3 bits free for out-of-band information.

Re: Weird things I learned while writing an x86 emulator

#45

Post by praptak »

m3kw9 wrote: post #43
woodruffw wrote: post #7

This is a good list! Another fun quirk: because x86 is a register-memory architecture and allows all kinds of variants of reg/mem operand encodings, there are a handful of equivalent encodings with exactly the same lengths (and just slightly different ModR/M bytes). You can take advantage of this to do software fingerprinting or, in my case, steganography without changing an executable’s size or semantics[1]. [1]: ht…

Why would I want to hide a message beside a program?

I think this has mostly curiosity/show off value. Encoding messages into executable binaries without changing their running behavior is kinda cool.

Re: Weird things I learned while writing an x86 emulator

#46

Post by a-priori »

amelius wrote: post #36

What resources did he use for this research? I would expect at least a reference to an Intel reference manual, but perhaps there are better ways to learn about the ISA.

There are reference manuals with a lot of details. Specifically you'd want to read the "Intel® 64 and IA-32 Architectures Software Developer’s Manual", Volumes 2, which includes specifications of each instruction including pseudo-code of how the processor executes them. You can freely access a PDF of the entire manual here: https://www.intel.com/content/www/us/en/developer/articles/t...

But there's a lot of details there — that volume alone spans about 4000 printed pages in the manual — so you're bound to mess things up.

I've written emulators for simpler instruction sets (RISC-V and the 6502) and found that the best way to ensure they work correctly is to do instruction-by-instruction comparisons against a reference. The good news is that if you've got a PC computer, you've got easy access to a reference machine! So you can do side-by-side comparisons where you set up an initial state on both a real machine and an emulated machine, then execute some code on both, and compare the final states. By using the trap flag you could also single-step through the instructions to extract the intermediate states after each instruction and ensure they match the emulated state.

So that's my guess about how he did his too. It's painstaking work, but also kind of fun when you get into it.

Re: Weird things I learned while writing an x86 emulator

#47

Post by woodruffw »

m3kw9 wrote: post #43
woodruffw wrote: post #7

This is a good list! Another fun quirk: because x86 is a register-memory architecture and allows all kinds of variants of reg/mem operand encodings, there are a handful of equivalent encodings with exactly the same lengths (and just slightly different ModR/M bytes). You can take advantage of this to do software fingerprinting or, in my case, steganography without changing an executable’s size or semantics[1]. [1]: ht…

Why would I want to hide a message beside a program?

It’s primarily a cute trick.

Re: Weird things I learned while writing an x86 emulator

#48

Post by dataflow »

userbinator wrote: post #13

An ADD instruction will update the carry flag but the INC instruction does not! Neither does DEC, and I believe the original reason for this was multiple-precision arithmetic routines --- you need to propagate the carry flag, but also update loop counters and pointers. It seems that the actual behavior of the undefined flags is related to the internal implementation of the shift operation, and is different between di…

> According to https://www.sandpile.org/x86/flags.htm which unfortunately hasn't been updated nor is exhaustive

Do you know if there's a more exhaustive source (besides the official manuals)?

Re: Weird things I learned while writing an x86 emulator

#49

Post by pm215 »

menaerus wrote: post #37
rcxdude wrote: post #30

Earlier quoted context omitted.

It can still debug multi-threaded programs, but if your bug relies on multi-core interactions it won't appear under rr (you can still catch a lot of race conditions though, and it has a 'chaos mode' which generate irregular scheduling intended to increase the likelyhood of such bugs appearing). It's been used to debug race conditions as well as other bugs in firefox, for example.

That's what I thought as well, limited to the single-core race conditions era. Thanks for the evidence.

It's still much better than the alternative, which is not having any kind of way to reproduce race conditions :-) QEMU's a pretty complex multithreaded program and my experience with chaos mode has been that it's quite good at tripping up the races (and you only need to catch the race once).

Re: Weird things I learned while writing an x86 emulator

#50

Post by jrockway »

menaerus wrote: post #39
woodruffw wrote: post #10

Earlier quoted context omitted.

It’s relatively feature complete, but there are some ideas for increasing the steganographic capacity listed in the README and issues! In particular, we could use the flexibility of the multi-byte NOP sequences to hide some more information.

What are some interesting use-cases for information hiding in the binaries?

Embed your customer's serial number in the binary at download time. When the file ends up on The Pirate Bay, send them a bill.
Post reply on HN