Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Things I learned while writing an x86 emulator (2023)

timdbg.com

Post reply on HN
Print view View original

101–110 of 135 posts

Re: Things I learned while writing an x86 emulator (2023)

#101

Post by corsix »

aengelke wrote: post #45
trealira wrote: post #9

Earlier quoted context omitted.

Studying the x86 architecture is kind of like studying languages with lots of irregularities and vestigial bits, and with competing grammatical paradigms, e.g. French. Other architectures, like RISC-V and ARMv8, are much more consistent.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

For an implementation of logical immediate encoding without the loop, see https://github.com/LuaJIT/LuaJIT/blob/04dca7911ea255f37be799...

Re: Things I learned while writing an x86 emulator (2023)

#102

Post by mrspuratic »

commandlinefan wrote: post #23
trollied wrote: post #20

> Writing a CPU emulator is, in my opinion, the best way to REALLY understand how a CPU works Hard disagree. The best way is to create a CPU from gate level, like you do on a decent CS course. (I really enjoyed making a cut down ARM from scratch)

OTOH, are you really going to be implementing memory segmenting in your gate-level CPU? I'd say actually creating a working CPU and _then_ emulating a real CPU (warts and all) are both necessary steps to real understanding.

This. I mean, why not start with wave theory and material science if you really want a good understanding :)

In my CS course I learned a hell of a lot from creating a 6800 emulator; though it wasn't on the course, building a working 6800 system was. The development involved running an assembler on a commercial *nix system and then typing the hex object code into an EPROM programmer. You get a lot of time to think about where your bugs are when you have to wait for a UV erase cycle...

Re: Things I learned while writing an x86 emulator (2023)

#103

Post by kimixa »

CoastalCoder wrote: post #18
aengelke wrote: post #10

Bonus quirk: there's BSF/BSR, for which the Intel SDM states that on zero input, the destination has an undefined value. (AMD documents that the destination is not modified in that case.) And then there's glibc, which happily uses the undocumented fact that the destination is also unmodified on Intel [1]. It took me quite some time to track down the issue in my binary translator. (There's also TZCNT/LZCNT, which is B…

Can you imagine having to make all this logic work faithfully, let alone fast , in silicon? X86 used to be Intel's moat, but what a nightmarish burden to carry.

It seems that the complexity of state management in a modern superscalar CPU is orders of magnitude more complex than even this.

Re: Things I learned while writing an x86 emulator (2023)

#104

Post by 256_ »

bheadmaster wrote: post #25
waynecochran wrote: post #16

Intel architecture is loaded with historical artifacts. The switch in how segment registers were used as you went from real mode to protected mode was an incredible hardware hack to keep older software working. I blame Intel for why so many folks avoid assembly language. I programmed in assembly for years using TI's 84010 graphics chips and the design was gorgeous -- simple RISC instruction set, flat address space, a…

If Terry A. Davis is to be trusted, as long as you ignore the legacy stuff, x64 assembly is nice to work with.

Where did he say this? In any case, I can confirm it from personal experience. x86 isn't that annoying if you ignore most of it. The worst thing is the edge cases with how some instructions can only use some registers, like shifting/rotating using cl.

Re: Things I learned while writing an x86 emulator (2023)

#105

Post by trealira »

jcranmer wrote: post #80
trealira wrote: post #61

Earlier quoted context omitted.

Admittedly, I never wrote an assembler, but the encoding of x86-64 seems pretty convoluted [0] [1], with much of the information smeared across the some bits in the Mod/RM and SIB bytes and then extended by prefix bytes. It's more complicated than you would assume from having only written assembly in text and then having the assembler encode the instructions. One of the things that sticks out to me is that on x86-64,…

One of the projects I work on every now and then is the "World's Worst X86 Decoder", where I'm trying to essentially automatically uncover x86 assembly semantics without ever having to build a list of x86 instructions, and this has forced me to look at the x86 ISA in a different way. The basic format I build for an opcode is this: enum Group1Prefix { Lock = 0xf0, Repnz = 0xf2, Repz = 0xf3 } enum Group2Prefix { Cs = 0…

That's pretty cool; thanks for showing me. I guess x86 instruction encoding isn't as bad as I thought, then.

Re: Things I learned while writing an x86 emulator (2023)

#106

Post by dzaima »

aengelke wrote: post #99
dzaima wrote: post #78

Earlier quoted context omitted.

A fun thing is that e.g. "cmp ax, 0x4231" differs from "cmp eax, 0x87654321" only in the presence of the data16 prefix, and thus the longer immediate; and it's the only significant case (I think?) of a prefix changing the total instruction size, and thus, for some such instructions, the 16-bit version, sometimes (but not always!) is significantly slower. "but not always" as in, if you try to microbenchmark a loop of…

The behavior/penalty of 66/67 length-changing prefixes is documented in the Intel Optimization Reference Manual (3.4.2.3): > Assembly/Compiler Coding Rule 19. (MH impact, MH generality) Favor generating code using imm8 or imm32 values instead of imm16 values.

Some interesting quotes around there:

> The following alignment situations can cause LCP stalls to trigger twice:

> · An instruction is encoded with a MODR/M and SIB byte, and the fetch line boundary crossing is between the MODR/M and the SIB bytes.

> · An instruction starts at offset 13 of a fetch line references a memory location using register and immediate byte offset addressing mode.

So that's the order of funkiness to be expected, fun.

> False LCP stalls occur when (a) instructions with LCP that are encoded using the F7 opcodes, and (b) are located at offset 14 of a fetch line. These instructions are: not, neg, div, idiv, mul, and imul. False LCP experiences delay because the instruction length decoder can not determine the length of the instruction before the next fetch line, which holds the exact opcode of the instruction in its MODR/M byte.

The "true" LCP stall for the F7 opcodes would be "test r16,imm16", but due to the split opcode info across the initial byte & ModR/M, the other F7's suffer too.

Re: Things I learned while writing an x86 emulator (2023)

#107

Post by brendank310 »

WalterBright wrote: post #68
aengelke wrote: post #45

Earlier quoted context omitted.

> Other architectures, like [...] ARMv8, are much more consistent. From an instruction/operation perspective, AArch64 is more clean. However, from an instruction operand and encoding perspective, AArch64 is a lot less consistent than x86. Consider the different operand types: on x86, there are a dozen register types, immediate (8/16/32/64 bits), and memory operands (always the same layout). On AArch64, there's: GP re…

As I'm currently implementing an AArch64 code generator for the D language dmd compiler, the inconsistency of its instructions is equaled and worsened by the clumsiness of the documentation for it :-/ But I'm slowly figuring it out. (For example, some instructions with very different encodings have the same mnemonic. Arrgghh.)

It's probably no longer maintained, but a former colleague of mine did some work on this for C++: https://github.com/ainfosec/shoulder. Obviously if the docs are lying it doesn't help much, but there was another effort he had https://github.com/ainfosec/scapula that tried to automate detecting behavior differences between the docs and the hardware implementation.

Re: Things I learned while writing an x86 emulator (2023)

#108

Post by commandlinefan »

mrspuratic wrote: post #102
commandlinefan wrote: post #23

Earlier quoted context omitted.

OTOH, are you really going to be implementing memory segmenting in your gate-level CPU? I'd say actually creating a working CPU and _then_ emulating a real CPU (warts and all) are both necessary steps to real understanding.

This. I mean, why not start with wave theory and material science if you really want a good understanding :) In my CS course I learned a hell of a lot from creating a 6800 emulator; though it wasn't on the course, building a working 6800 system was. The development involved running an assembler on a commercial *nix system and then typing the hex object code into an EPROM programmer. You get a lot of time to think abo…

> why not start with wave theory and material science

You jest, but if I had infinite time...

Re: Things I learned while writing an x86 emulator (2023)

#109

Post by adrian_b »

bonzini wrote: post #74
mananaysiempre wrote: post #38

Earlier quoted context omitted.

The semantics of LZCNT combined with its encoding feels like an own goal: it’s encoded as a BSR instruction with a legacy-ignored prefix, but for nonzero inputs its return value is the operand size minus the return value of the legacy version. Yes, clz() is a function that exists, but the extra subtraction in its implementation feels like a small cost to pay for extra compatibility when LZCNT could’ve just been BSR w…

Yes, it's like someone looked at TZCNT and thought "let's encode LZCNT the same way", but it makes no sense.

LZCNT and TZCNT are corrections (originally introduced by AMD) for the serious mistake done by the designers of Intel 80386 when they have defined BSF and BSR.

Because on the very slow 80386 the wrong definition for the null input did not matter much, they have failed to foresee how bad it will become for the future pipelined and superscalar CPUs, where having to insert a test for null input can slow down a program many times.

Nevertheless, they should have paid more attention to the earlier use of such instructions. For instance Cray-1 had defined LZCNT in the right way almost ten years earlier.

Re: Things I learned while writing an x86 emulator (2023)

#110

Post by mananaysiempre »

BeeOnRope wrote: post #94
mananaysiempre wrote: post #38

Earlier quoted context omitted.

The semantics of LZCNT combined with its encoding feels like an own goal: it’s encoded as a BSR instruction with a legacy-ignored prefix, but for nonzero inputs its return value is the operand size minus the return value of the legacy version. Yes, clz() is a function that exists, but the extra subtraction in its implementation feels like a small cost to pay for extra compatibility when LZCNT could’ve just been BSR w…

I'm not following: as long as you are introducing a new, incompatible instruction for leading zero counting, you'd definitely choose LZCNT over BSR as LZCNT has definitely won in retrospect over BSR as the primitive for this use case. BSR is just a historical anomaly which has a zero-input problem for no benefit. What would be the point of offering a new variation BSR with different input semantics?

When it comes to TZCNT vs BSF, they are just compatible enough for a new compiler to use unconditionally (if we assume that BSF with a zero input leaves its output register unchanged, as it has for decades, and as documented by AMD who defined LZCNT): the instruction sequence

  MOV   ECX, 32
  TZCNT ECX, EAX ; i.e. REP BSF ECX, EAX
behaves identically on everything from the original 80386 up and is better on superscalars with TZCNT support due to avoiding the false dependency on ECX. The reason for that is that BSF with a nonzero input and TZCNT with a nonzero input have exactly the same output. That’s emphatically not true of BSR and LZCNT, so we’re stuck relegating LZCNT to compiler flags.
Post reply on HN