Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned while writing an x86 emulator (2023)

timdbg.com

Post reply on HN
Print view View original

61–70 of 135 posts

Re: Things I learned while writing an x86 emulator (2023)

#61

Post by trealira »

aengelke wrote: post #45
trealira wrote: post #9

Earlier quoted context omitted.

Studying the x86 architecture is kind of like studying languages with lots of irregularities and vestigial bits, and with competing grammatical paradigms, e.g. French. Other architectures, like RISC-V and ARMv8, are much more consistent.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions.

One of the things that sticks out to me is that on x86-64, operations on 32-bit registers implicitly zero out the upper 32-bits of the corresponding 64-bit register (e.g. "MOV EAX, EBX" also zeroes out the upper 32-bits of RAX), except for the opcode 0x90, which, logically, would be the accumulator encoding of "XCHG EAX, EAX" but is special-cased to do nothing because it's the traditional NOP instruction for x86. So you have to use the other encoding, 87 C0.

The fact that AArch64 has instructions that are almost always 4 bytes, and the fact that they clearly thought out their instruction set more (e.g. instead of having CMP, they just use SUBS (subtract and store condition codes) and store the result in the zero register, which is always zero), is what made me say that it seemed more regular. I hadn't studied it in as close detail as x86-64, though. But, you've written an an ARM disassembler and I haven't; you seem to know more about it than me, and so I believe what you're saying.

> AArch64 also has some register constraints

x86-64 also has some, in that many instructions can't encode the upper 8 bits of a 16-bit register (AH, BH, DH, CH) if a REX prefix is used.

[0]: https://wiki.osdev.org/X86-64_Instruction_Encoding

[1]: http://www.c-jump.com/CIS77/CPU/x86/lecture.html

Re: Things I learned while writing an x86 emulator (2023)

#62

Post by brailsafe »

trollied wrote: post #20

> Writing a CPU emulator is, in my opinion, the best way to REALLY understand how a CPU works Hard disagree. The best way is to create a CPU from gate level, like you do on a decent CS course. (I really enjoyed making a cut down ARM from scratch)

So far on my journey through Nand2Tetris (since I kind of dropped out of my real CS course) I've found the entire process of working my way up from gate level, and just finished the VM emulator chapter which took an eternity. Now onto compilation.

Re: Things I learned while writing an x86 emulator (2023)

#63

Post by banish-m4 »

trollied wrote: post #20

> Writing a CPU emulator is, in my opinion, the best way to REALLY understand how a CPU works Hard disagree. The best way is to create a CPU from gate level, like you do on a decent CS course. (I really enjoyed making a cut down ARM from scratch)

Seconded. A microcoded, pipelined, superscalar, branch-predicting basic processor with L1 data & instruction caches and write-back L2 cache controller is nontrivial. Most software engineers have an incomplete grasp of data hazards, cache invalidation, or pipeline stalls.

Re: Things I learned while writing an x86 emulator (2023)

#64

Post by banish-m4 »

timmisiak wrote: post #54
trollied wrote: post #20

> Writing a CPU emulator is, in my opinion, the best way to REALLY understand how a CPU works Hard disagree. The best way is to create a CPU from gate level, like you do on a decent CS course. (I really enjoyed making a cut down ARM from scratch)

I think both are useful, but designing a modern CPU from the gate level is out of reach for most folks, and I think there's a big gap between the sorts of CPUs we designed in college and the sort that run real code. I think creating an emulator of a modern CPU is a somewhat more accessible challenge, while still being very educational even if you only get something partially working.

This is an illusion and a red herring. RTL synthesis is the typical functional prototype stage reached which is generally sufficient for FPGA work. To burn an ASIC as part of an educational consortium run is doable, but it's uncommon.

Re: Things I learned while writing an x86 emulator (2023)

#65

Post by jart »

aengelke wrote: post #42
bonzini wrote: post #33

Earlier quoted context omitted.

On and off over the last year I have been rewriting QEMU's x86 decoder. It started as a necessary task to incorporate AVX support, but I am now at a point where only a handful of opcodes are left to rewrite, after which it should not be too hard to add APX support. For EVEX my plan is to keep the raw bits until after the opcode has been read (i.e. before immediates and possibly before modrm) and the EVEX class identi…

Thanks for the pointer to QEMU's decoder! I actually never looked at it before. So you coded all the tables manually in C -- interesting, that's quite some effort. I opted to autogenerate the tables (and keep them as data only => smaller memory footprint) [1,2]. That's doable, because x86 encodings are mostly fairly consistent. I can also generate an encoder from it (ok, you don't need that). Re 'custom size "xh"': A…

FWIW here's all 700 lines of Blink's x86 decoder. https://github.com/jart/blink/blob/master/blink/x86.c

Re: Things I learned while writing an x86 emulator (2023)

#66

Post by banish-m4 »

fjfaase wrote: post #26

Interesting read. I have a lot of respect for people who develop emulator for x86 processors. It is a complicated processor and from first hand experience I know that developing and debugging emulators for CPU's can be very challenging. In the past year, I spend some time developing a very limited i386 emulator [1] including some system calls for executing the first steps of live-bootstrap [2], primarily to figure ou…

Most of the complexities lie in managing the various configurations of total system compatibility emulation, especially for timing, analog oddities, and whether to include bugs or not and for which steppings. If you want precise and accurate emulation, you have to have real hardware to validate behavior against. Then there are the cases of what not to emulate and offering better-than-original alternatives.

Re: Things I learned while writing an x86 emulator (2023)

#67

Post by WalterBright »

timmisiak wrote: post #54
trollied wrote: post #20

> Writing a CPU emulator is, in my opinion, the best way to REALLY understand how a CPU works Hard disagree. The best way is to create a CPU from gate level, like you do on a decent CS course. (I really enjoyed making a cut down ARM from scratch)

I think both are useful, but designing a modern CPU from the gate level is out of reach for most folks, and I think there's a big gap between the sorts of CPUs we designed in college and the sort that run real code. I think creating an emulator of a modern CPU is a somewhat more accessible challenge, while still being very educational even if you only get something partially working.

When I was at Caltech, another student in the dorm had been admitted because he'd designed and implemented a CPU using only 7400 TTL.

Woz wasn't the only supersmart young computer guy at the time :-)

(I don't know how capable it was, even a 4 bit CPU would be quite a challenge with TTL.)

Re: Things I learned while writing an x86 emulator (2023)

#68

Post by WalterBright »

aengelke wrote: post #45
trealira wrote: post #9

Earlier quoted context omitted.

Studying the x86 architecture is kind of like studying languages with lots of irregularities and vestigial bits, and with competing grammatical paradigms, e.g. French. Other architectures, like RISC-V and ARMv8, are much more consistent.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

As I'm currently implementing an AArch64 code generator for the D language dmd compiler, the inconsistency of its instructions is equaled and worsened by the clumsiness of the documentation for it :-/ But I'm slowly figuring it out.

(For example, some instructions with very different encodings have the same mnemonic. Arrgghh.)

Re: Things I learned while writing an x86 emulator (2023)

#69

Post by WalterBright »

trealira wrote: post #61
aengelke wrote: post #45

Earlier quoted context omitted.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions. One of the things that sticks out to me is that on x86-64,…

> you've written an an ARM disassembler

Here's my AArch64 disassembler work in progress:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ba...

I add to it in tandem with writing the code generator. It helps flush out bugs in both by doing this. I.e. generate the instruction, the disassemble it and compare with what I thought it should be.

It's quite a bit more complicated than the corresponding x86 disassembler:

https://github.com/dlang/dmd/blob/master/compiler/src/dmd/ba...

Re: Things I learned while writing an x86 emulator (2023)

#70

Post by boricj »

It's funny to me how much grief x86 assembly generates when compared to RISC here, because I have the opposite problem when delinking code back into object files.

For this use-case, x86 is really easy to analyze whereas MIPS has been a nightmare to pull off. This is because all I mostly care about are references to code and data. x86 has pointer-sized immediate constants and MIPS has split HI16/LO16 relocation pairs, which leads to all sorts of trouble with register usage graphs, code flow and branch delay instructions.

That should not be constructed as praise on my end for x86.

Post reply on HN