Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned while writing an x86 emulator (2023)

timdbg.com

Post reply on HN
Print view View original

71–80 of 135 posts

Re: Things I learned while writing an x86 emulator (2023)

#71

Post by WalterBright »

x0x0 wrote: post #17
trealira wrote: post #9

Earlier quoted context omitted.

Studying the x86 architecture is kind of like studying languages with lots of irregularities and vestigial bits, and with competing grammatical paradigms, e.g. French. Other architectures, like RISC-V and ARMv8, are much more consistent.

I think English may be a better example; we just stapled chunks of vulgar latin to an inconsistently simplified proto-germanic and then borrowed words from every language we met along the way. Add in 44 sounds serialized to the page with 26 letters and tada!

The Norman conquest of England means English is a language with barbarian syntax and French nouns. It's a happy mess of cultural appropriation!

Squeezing the lot into 26 characters was simply genius - enabling printing with movable type, Morse code, Baudot code, ASCII, etc.

Of course, then icons came along and ruined everything.

Re: Things I learned while writing an x86 emulator (2023)

#72

Post by [deleted] »

jcranmer wrote: post #51
snazz wrote: post #35

Earlier quoted context omitted.

What sorts of projects are you working on that use Itanium?

None, really. I just happened to get a copy of the manual and start idly reading it when my computer got stuck in a very long update-reboot cycle and I couldn't do anything other than read a physical book.

[deleted]

Re: Things I learned while writing an x86 emulator (2023)

#73

Post by bonzini »

jart wrote: post #65
aengelke wrote: post #42

Earlier quoted context omitted.

Thanks for the pointer to QEMU's decoder! I actually never looked at it before. So you coded all the tables manually in C -- interesting, that's quite some effort. I opted to autogenerate the tables (and keep them as data only => smaller memory footprint) [1,2]. That's doable, because x86 encodings are mostly fairly consistent. I can also generate an encoder from it (ok, you don't need that). Re 'custom size "xh"': A…

FWIW here's all 700 lines of Blink's x86 decoder. https://github.com/jart/blink/blob/master/blink/x86.c

I don't want to be the person that has to add an instruction to blink...

Re: Things I learned while writing an x86 emulator (2023)

#74

Post by bonzini »

mananaysiempre wrote: post #38
aengelke wrote: post #10

Bonus quirk: there's BSF/BSR, for which the Intel SDM states that on zero input, the destination has an undefined value. (AMD documents that the destination is not modified in that case.) And then there's glibc, which happily uses the undocumented fact that the destination is also unmodified on Intel [1]. It took me quite some time to track down the issue in my binary translator. (There's also TZCNT/LZCNT, which is B…

The semantics of LZCNT combined with its encoding feels like an own goal: it’s encoded as a BSR instruction with a legacy-ignored prefix, but for nonzero inputs its return value is the operand size minus the return value of the legacy version. Yes, clz() is a function that exists, but the extra subtraction in its implementation feels like a small cost to pay for extra compatibility when LZCNT could’ve just been BSR w…

Yes, it's like someone looked at TZCNT and thought "let's encode LZCNT the same way", but it makes no sense.

Re: Things I learned while writing an x86 emulator (2023)

#75

Post by bonzini »

trealira wrote: post #61
aengelke wrote: post #45

Earlier quoted context omitted.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions. One of the things that sticks out to me is that on x86-64,…

x86 has a lot of special cases and it's becoming harder and harder to find non-code material with all instructions in a nice tabular format. But overall the encoding isn't that complicated. What's complex in the architecture is the amount of legacy stuff that still lives in privileged code.

> x86-64 also has some, in that many instructions can't encode the upper 8 bits of a 16-bit register (AH, BH, DH, CH) if a REX prefix is used.

You can't use both the high registers and the extended low registers in the same instruction, indeed. But in practice nobody is using the high registers anymore so it's a relatively rare limitation.

Re: Things I learned while writing an x86 emulator (2023)

#76

Post by jcranmer »

bonzini wrote: post #73
jart wrote: post #65

Earlier quoted context omitted.

FWIW here's all 700 lines of Blink's x86 decoder. https://github.com/jart/blink/blob/master/blink/x86.c

I don't want to be the person that has to add an instruction to blink...

It looks like someone started with Intel's XED code (which relies on custom tables to specify instructions, and compiles that to C tables at compile time) and hand-minimized the code into a single file. I'm guessing it's designed to never have any more code added to it.

Re: Things I learned while writing an x86 emulator (2023)

#77

Post by moosedev »

dx4100 wrote: post #22
CoastalCoder wrote: post #18

Earlier quoted context omitted.

Can you imagine having to make all this logic work faithfully, let alone fast , in silicon? X86 used to be Intel's moat, but what a nightmarish burden to carry.

Did people just... do this by hand (in software), transistor by transistor, or was it laid out programmatically in some sense? As in, were segments created algorithmically, then repeated to obtain the desired outcome? CPU design baffles me, especially considering there are 134 BILLION transistors or so in the latest i7 CPU. How does the team even keep track of, work on, or even load the files to WORK on the CPUs?

You might enjoy Ken’s latest article on some of this stuff, posted just the other day: https://news.ycombinator.com/item?id=40899393

Re: Things I learned while writing an x86 emulator (2023)

#78

Post by dzaima »

CoastalCoder wrote: post #18
aengelke wrote: post #10

Bonus quirk: there's BSF/BSR, for which the Intel SDM states that on zero input, the destination has an undefined value. (AMD documents that the destination is not modified in that case.) And then there's glibc, which happily uses the undocumented fact that the destination is also unmodified on Intel [1]. It took me quite some time to track down the issue in my binary translator. (There's also TZCNT/LZCNT, which is B…

Can you imagine having to make all this logic work faithfully, let alone fast , in silicon? X86 used to be Intel's moat, but what a nightmarish burden to carry.

A fun thing is that e.g. "cmp ax, 0x4231" differs from "cmp eax, 0x87654321" only in the presence of the data16 prefix, and thus the longer immediate; and it's the only significant case (I think?) of a prefix changing the total instruction size, and thus, for some such instructions, the 16-bit version, sometimes (but not always!) is significantly slower. "but not always" as in, if you try to microbenchmark a loop of such, sometimes you can have entire microseconds of it consistently running at 0.25 cycles/instr avg, and sometimes that same exact code (in the same process!) will measure it at 3 cycles/instr (tested on Haswell, but uops.info indicates this happens on all non-atom Intel since Ivy Bridge).

Re: Things I learned while writing an x86 emulator (2023)

#79

Post by jmspring »

Apparently my memory is false, I thought originally the salsa20 variants and machine code were on cryp.to in my memory, but Dan Berstein's site is - https://cr.yp.to/

While at a startup when we were looking at data at rest encryption, streaming encryption and other such things. Dan had a page with different implementations (cross compiled from his assembler representation) to target chipsets and instruction sets. Using VMs (this was the early/mid 2000s) and such, it was interesting to see what of those instruction sets were supported. In testing, there would be occasional hiccups where an implementation wasn't fully supported though the VM claimed such.

Re: Things I learned while writing an x86 emulator (2023)

#80

Post by jcranmer »

trealira wrote: post #61
aengelke wrote: post #45

Earlier quoted context omitted.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions. One of the things that sticks out to me is that on x86-64,…

One of the projects I work on every now and then is the "World's Worst X86 Decoder", where I'm trying to essentially automatically uncover x86 assembly semantics without ever having to build a list of x86 instructions, and this has forced me to look at the x86 ISA in a different way. The basic format I build for an opcode is this:

  enum Group1Prefix { Lock = 0xf0, Repnz = 0xf2, Repz = 0xf3 }
  enum Group2Prefix { Cs = 0x2e, Ds = 0x3e, Es = 0x26, Fs = 0x64, Gs = 0x65, Ss = 0x36 }
  enum Group3Prefix { OpSize = 0x66 }
  enum Group4Prefix { AddrSize = 0x67 }
  enum ModernPrefix {
    Rex { w: bool },
    Vex { w: bool, l: bool },
  }
  struct Opcode {
    pub group1: Option,
    pub group2: Option,
    pub group3: Option,
    pub group4: Option,
    pub modern_prefix: Option,
    pub opcode_map: u8,
    pub opcode: u8,
  }
And all opcodes can optionally have an immediate of 1, 2, 3, 4, or 8 bytes and optionally have a ModR/M byte (which is a separate datastructure because I don't enter that information myself, I simply run a sandsifter-like program to execute every single opcode and work out the answer).

This isn't quite accurate to how an assembler would see it, as Intel will sometimes pack instructions with 0 or 1 operands into a ModR/M byte, so that, e.g., 0F.01 eax, [mem] is actually the SGDT instruction and 0F.01 ecx, [mem] is SIDT (and 0F.01 eax, ecx is actually VMCALL).

As long as you're internally making a distinction between the different operand forms of instructions (e.g., 8-bit ADD versus 16-bit versus 32-bit versus 64-bit, and register/register versus register/immediate versus register/memory), it's actually not all that difficult to deal with the instruction encoding, or even mapping IR to those instructions, at least until EVEX prefixes enter the picture.

Post reply on HN