Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fundamental flaws of SIMD ISAs (2021)

bitsnbites.eu

Post reply on HN
Print view View original

121–130 of 146 posts

Re: Fundamental flaws of SIMD ISAs (2021)

#121

Post by dzaima »

timewizard wrote: post #33

> Another problem is that each new SIMD generation requires new instruction opcodes and encodings. It requires new opcodes. It does not strictly require new encodings. Several new encodings are legacy compatible and can encode previous generations vector instructions. > so the architecture must provide enough SIMD registers to avoid register spilling. Or the architecture allows memory operands. The great joy of basic…

> The great joy of basic x86 encoding is that you don't actually need to put things in registers to operate on them.

That's... 1 register saved, out of 16 (or 32 on AVX-512). Perhaps useful sometimes, but far from a particularly significant aspect spill-wise.

And doing that means you lose the ability to move the load earlier (perhaps not too significant on OoO hardware, but still a consideration; while reorder windows are multiple hundreds of instructions, the actual OoO limit is scheduling queues, which are frequently under a hundred entries, i.e. a couple dozen cycles worth of instructions, at which point the ≥4 cycle latency of a load is not actually insignificant. And putting the load directly in the arith op is the worst-case scenario for this)

Re: Fundamental flaws of SIMD ISAs (2021)

#122

Post by dzaima »

convolvatron wrote: post #3

i would certainly add lack of reductions ('horizontal' operations) and a more generalized model of communication to the list.

ARM NEON does have sum, min, and max reductions (and/or reductions can just be min/max if all bits in elements are the same), along with pairwise ops. RVV has sum,min,max,and,or,xor reductions. x86 has psadbw which sums windows of eight 8-bit ints, and various instructions for some pairwise horizontal stuff.

But in general building code around reductions isn't really a thing you'd ideally do; they're necessarily gonna have higher latency / lower throughput / take more silicon compared to avoiding them where possible, best to leave reducing to a single element to the loop tail.

Re: Fundamental flaws of SIMD ISAs (2021)

#123

Post by Remnant44 »

gpderetta wrote: post #99
sweetjuly wrote: post #10

Loop unrolling isn't really done because of pipelining but rather to amortize the cost of looping. Any modern out-of-order core will (on the happy path) schedule the operations identically whether you did one copy per loop or four. The only difference is the number of branches.

The looping overhead is trivial, especially on simd code where the loop overhead will use the scalar hardware. Unrolling is definitely needed for properly scheduling and pipelining SIMD code even on OoO cores. Remember that an OoO core cannot reorder dependent instructions, so the dependencies need to be manually broken, for example by adding additional accumulators, which in turn requires additional unrolling, this…

That's a good point about increased dependency chain length in simd due to the branchless programming style. Unrolling to break a loop-carried dependency is one of the strongest reasons to unroll especially simd code.

Unrolling trivial loops to remove loop counter overhead hasn't been productive for quite a whole now but unfortunately it's still the default for many compilers.

Re: Fundamental flaws of SIMD ISAs (2021)

#124

Post by kevingadd »

Someone wrote: post #102
__abadams__ wrote: post #55

Earlier quoted context omitted.

It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this. Clang does it well if you use its built-in vector types (which can be any length). Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you just…

> It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this Wouldn’t that be suboptimal if/when CPUs that support 1024-bit vectors come along? > Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you j…

Optimal performance in a vector algorithm typically requires optimizing around things like the number of available registers, whether the registers in use are volatile (mandating stack spills when calling other functions like a comparer), and sizes of sequences.

If you know you're engineering for 16-byte vectors you can 'just' align all your data to 16 bytes. And if you know you have 8 vector registers where 4 of them are non-volatile you can design around that too. But without information like that you have to be defensive, like aligning all your data to 128 bytes instead Just In Case (heaven forbid native vectors get bigger than that), minimizing the number of registers you use to try and avoid stack spills, etc. (I mention this because WASM also doesn't expose any of this information.)

It's true that you could just design for a static size on a system with variable-length vectors. I suspect you'd see a lot of people do that, and potentially under-utilize the hardware's capabilities. Better than nothing, at least!

Re: Fundamental flaws of SIMD ISAs (2021)

#125

Post by dzaima »

codedokode wrote: post #83

I think that packed SIMD is better in almost every aspect and Vector SIMD is worse. With vector SIMD you don't know the register size beforehand and therefore have to maintain and increment counters, adding extra unnecessary instructions, reducing total performance. With packed SIMD you can issue several loads immediately without dependencies, and if you look at code examples, you can see that the x86 code is more de…

> Open source software is recompiled every week anyway.

Despite being potentially compiled recently, anything from most Linux package managers, and whatever precompiled downloadable executables, even if from open-source code, still targets the 20-year-old SSE2 baseline, wasting the majority of SIMD resources available on modern (..or just not-extremely-ancient) CPUs (unless you're looking at the 0.001% of software that bothers with dynamic dispatch; but for that approach just recompiling isn't enough, you also need to extend the dispatched target set).

RISC-V RVV's LMUL means that you get unrolling for free, as each instruction can operate over up to 8 registers per operand, i.e. essentially "hardware 8x unrolling", thereby making scalar overhead insignificant. (probably a minor nightmare from the silicon POV, but perhaps not in a particularly limiting way - double-pumping has been done by x86 many times so LMUL=2 is simple enough, and at LMUL=4 and LMUL=8 you can afford to decode/split into ups at 1 instr/cycle)

ARM SVE can encode adding a multiple of VL in load/store instructions, allowing manual unrolling without having to actually compute the intermediate sizes. (hardware-wise that's an extremely tiny amount of overhead, as it's trivially mappable to an immediate offset at decode time). And there's an instruction to bump a variable by a multiple of VL.

And you need to bump pointers in any SIMD regardless; the only difference is whether the bump size is an immediate, or a dynamic value, and if you control the ISA you can balance between the two as necessary. The packed SIMD approach isn't "free" either - you need hardware support for immediate offsets in SIMD load/store instrs.

Even in a hypothetical non-existent bad vector SIMD ISA without any applicable free offsetting in loads/stores and a need for unrolling, you can avoid having a dependency between unrolled iterations by precomputing "vlen*2", "vlen*3", "vlen*4", ... outside of the loop and adding those as necessary, instead of having a strict dependency chain.

Re: Fundamental flaws of SIMD ISAs (2021)

#126

Post by ack_complete »

ezekiel68 wrote: post #95
ack_complete wrote: post #66

Earlier quoted context omitted.

AVX-512 also has a lot of wonderful facilities for autovectorization, but I suspect its initial downclocking effects plus getting yanked out of Alder Lake killed a lot of the momentum in improving compiler and library usage of it. Even the Steam Hardware Survey, which is skewed toward upper end hardware, only shows 16% availability of baseline AVX-512, compared to 94% for AVX2.

You mentioned "initial downclocking effects", yet (for posterity) I want to emphasize that in 2020 Ice Lake (Sunny Cove core) and later Intel processors, the downclocking is really a nothingburger. The fusing off debacle in desktop CPU families like Alder Lake you mentioned definitely killed the momentum though. I'm not sure why OS kernels couldn't have become partners in CPU capability queries (where a program start…

Sure, but any core-wide downclocking effect at all is annoying for autovectorization, since a small local win easily turns into a global loss. Which is why compilers have "prefer vector width" tuning parameters so autovec can be tuned down to avoid 512-bit or even 256-bit ops.

This is also the same reason that having AVX-512 only on the P-cores wouldn't have worked, even with thread director support. It would only take one small routine in a common location to push most threads off the P-cores.

I'm of the opinion that Intel's hybrid P/E-arch has been mostly useless anyway and only good for winning benchmarks. My current CPU has a 6P4E configuration and the scheduler hardly uses the E-cores at all unless forced, plus performance was better and more stable with the E-cores disabled.

Re: Fundamental flaws of SIMD ISAs (2021)

#127

Post by almostgotcaught »

pornel wrote: post #118
almostgotcaught wrote: post #112

Earlier quoted context omitted.

i have no idea what you're saying - i'm well aware that compilers do lots of things but this sentence in your original comment > compiled machine code exactly as-is, and had a instruction-updating pass implies there should be silicon that implements the instruction-updating - what else would be "executing" compiled machine code other than the machine itself...........

I was talking about a software pass. Currently, the machine code stored in executables (such as ELF or PE) is only slightly patched by the dynamic linker, and then expected to be directly executable by the CPU. The code in the file has to be already compatible with the target CPU, otherwise you hit illegal instructions. This is a simplistic approach, dating back to when running executables was just a matter of loadin…

that's certainly more reasonable so i'm sorry for being so flippant. but even this idea i wager the juice is not worth the squeeze outside of stuff like Rosetta as you alluded, where the value was extremely high (retaining x86 customers).

Re: Fundamental flaws of SIMD ISAs (2021)

#128

Post by ack_complete »

whizzter wrote: post #97
adgjlsfhk1 wrote: post #81

Earlier quoted context omitted.

> For instance, Skymont finally has no penalty for denormals, a long standing Intel weakness. yeah, that's crazy to me. Intel has been so completely discunctional for the last 15 years. I feel like you couldn't have a clearer sign of "we have 2 completely separate teams that are competing with each other and aren't allowed to/don't want to talk to each other". it's just such a clear sign that the chicken is running a…

Not really, to me it more seems like Pentium-4 vs Pentium-M/Core again. The downfall of Pentium 4 was that they had been stuffing things into longer and longer pipes to keep up the frequency race(with horrible branch latencies as a result). They scaled it all away by "resetting" to the P3/P-M/Core architecture and scaling up from that again. Pipes today are even _longer_ and if E-cores has shorter pipes at a similar…

Thankfully, the P-cores are nowhere near as bad as the Pentium 4 was. The Pentium 4 had such a skewed architecture that it was annoyingly frustrating to optimize for. Not only was the branch misprediction penalty long, but all common methods of doing branchless logic like conditional moves were also slow. It also had a slow shifter such that small left shifts were actually faster as sequences of adds, which I hadn't needed to do since the 68000 and 8086. And an annoying L1 cache that had 64K aliasing penalties (guess which popular OS allocates all virtual memory, particularly thread stacks, at 64K alignment.....)

The P-cores have their warts, but are still much more well-rounded than the P4 was.

Re: Fundamental flaws of SIMD ISAs (2021)

#129

Post by ack_complete »

the__alchemist wrote: post #113
PaulHoule wrote: post #28

Earlier quoted context omitted.

In AVX-512 we have a platform that rewards the assembly language programmer like few platforms have since the 6502. I see people doing really clever things that are specific to the system and one level it is really cool but on another level it means SIMD is the domain of the specialist, Intel puts out press releases about the really great features they have for the national labs and for Facebook whereas the rest of u…

Noob question! What about AVX-512 makes it unique to assembly programmers? I'm just dipping my toes in, and have been doing some chemistry computations using f32x8, Vec3x8 etc (AVX-256). I have good workflows set up, but have only been getting 2x speedup over non-SIMD. (Was hoping for closer to 8). I figured AVX-512 would allow f32x16 etc, which would be mostly a drop-in. (I have macros to set up the types, and you i…

AVX-512 has a lot of instructions that just extend vectorization to 512-bit and make it nicer with features like masking. Thus, a very valid use of it is just to double vectorization width.

But it also has a bunch of specialized instructions that can boost performance beyond just the 2x width. One of them is VPCOMPRESSB, which accelerates compact encoding of sparse data. Another is GF2P8AFFINEQB, which is targeted at specific encryption algorithms but can also be abused for general bit shuffling. Algorithms like computing a histogram can benefit significantly, but it requires reshaping the algorithm around very particular and peculiar intermediate data layouts that are beyond the transformations a compiler can do. This doesn't literally require assembly language, though, it can often be done with intrinsics.

Re: Fundamental flaws of SIMD ISAs (2021)

#130

Post by dataflow »

Someone wrote: post #102
__abadams__ wrote: post #55

Earlier quoted context omitted.

It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this. Clang does it well if you use its built-in vector types (which can be any length). Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you just…

> It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this Wouldn’t that be suboptimal if/when CPUs that support 1024-bit vectors come along? > Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you j…

> Wouldn’t that be suboptimal if/when CPUs that support 1024-bit vectors come along?

Is that likely or on anyone's roadmap? It makes a little less sense than 512 bits, at least for Intel, since their cache lines are 64 bytes i.e. 512 bits. Any more than that and they'd have to mess with multiple cache lines all the time, not just on unaligned accesses. And they'd have to support crossing more than 2 cache lines on unaligned accesses. They increase the cache line size too, but that seems terrible for compatibility since a lot of programs assume it's a compile time constant (and it'd have performance overhead to make it a run-time value). Somehow it feels like this isn't the way to go, but hey, I'm not a CPU architect.

Post reply on HN