Fundamental flaws of SIMD ISAs (2021) • Hacker News Board

Re: Fundamental flaws of SIMD ISAs (2021)

#61

Post by gatane » Fri, Apr 25, 2025, 12:37 AM UTC

PaulHoule wrote: post #52

bee_rider wrote: post #50

Earlier quoted context omitted.

It is kind of a bummer that MKL isn’t open sourced, as that would make inclusion in Linux easier. It is already free-as-in-beer, but of course that doesn’t solve everything. Baffling that MS didn’t use it. They have a pretty close relationship… Agree that they are sort of going after hard-to-use niche features nowadays. But I think it is just that the real thing we want—single threaded performance for branchy code—is…

At the very least you can decode UTF-8 really quickly with AVX-512 https://lemire.me/blog/2023/08/12/transcoding-utf-8-strings-... and web browsers at the very least spent a lot of cycles on decoding HTML and Javascript which is UTF-8 encoded. It turns out AVX-512 is good at a lot of things you wouldn't think SIMD would be good at. Intel's got the problem that people don't want to buy new computers because they don't…

In the end, it doesnt even matter, javascript frameworks are already big enough to slow down your pc.

Unless if said optimization on parsing runs at the very core of JS.

Re: Fundamental flaws of SIMD ISAs (2021)

#62

Post by dragontamer » Fri, Apr 25, 2025, 1:05 AM UTC

dzaima wrote: post #47

dragontamer wrote: post #45

1. Not a problem for GPUs. NVdia and AMD are both 32-wide or 1024-bit wide hard coded. AMD can swap to 64-wide mode for backwards compatibility to GCN. 1024-bit or 2048-bit seems to be the right values. Too wide and you get branch divergence issues, so it doesn't seem to make sense to go bigger. In contrast, the systems that have flexible widths have never taken off. It's seemingly much harder to design a programming…

Tail handling is not significant for loops with tons of iterations, but there are a ton of real-world situations where you might have a loop take only like 5 iterations or something (even at like 100 iterations, with a loop processing 8 elements at a time (i.e. 256-bit vectors, 32-bit elements), that's 12 vectorized iterations plus up to 7 scalar ones, which is still quite significant. At 1000 iterations you could st…

Note that AVX512 have per-lane execution masks so I'm not fully convinced that tail handling should even be a thing anymore.

If(my lane is beyond the buffer) then (exec flag off, do not store my lane).

Which in practice should be a simple vcompress instruction (AVX512 register) and maybe a move afterwards??? I admit that I'm not an AVX512 expert but it doesn't seem all that difficult with vcompress instructions + execmask.

Re: Fundamental flaws of SIMD ISAs (2021)

#63

Post by bjourne » Fri, Apr 25, 2025, 1:19 AM UTC

aengelke wrote: post #13

TinkersW wrote: post #4

I write a lot of SIMD and I don't really agree with this.. Flaw1:fixed width I prefer fixed width as it makes the code simpler to write, size is known as compile time so we know the size of our structures. Swizzle algorithms are also customized based on the size. Flaw2:pipelining no CPU I care about is in order so mostly irrelevant, and even scalar instructions are pipelined Flaw3: tail handling I code with SIMD as t…

I agree; and the article seems to have also quite a few technical flaws: - Register width: we somewhat maxed out at 512 bits, with Intel going back to 256 bits for non-server CPUs. I don't see larger widths on the horizon (even if SVE theoretically supports up to 2048 bits, I don't know any implementation with ~~>256~~ >512 bits). Larger bit widths are not beneficial for most applications and the few applications tha…

> "Loop unrolling also increases register pressure" -- it does, but code that really requires >32 registers is extremely rare, so a good instruction scheduler in the compiler can avoid spilling.

No, it actually is super common in hpc code. If you unroll a loop N times you need N times as many registers. For normal memory-bound code I agree with you, but most hpc kernels will exploit as much of the register file as they can for blocking/tiling.

Re: Fundamental flaws of SIMD ISAs (2021)

#64

Post by mattst88 » Fri, Apr 25, 2025, 1:19 AM UTC

whaleofatw2022 wrote: post #44

luyu_wu wrote: post #39

Earlier quoted context omitted.

Low power RISC cores (both ARM and RISC-V) are typically in-order actually! But any core I can think of as 'high-performance' is OOO.

MIPS as well as Alpha AFAIR. And technically itanium, otoh It seems to me a bit like a niche for any performance advantages...

Alpha 21264 is out-of-order.

Re: Fundamental flaws of SIMD ISAs (2021)

#65

Post by ack_complete » Fri, Apr 25, 2025, 1:39 AM UTC

adgjlsfhk1 wrote: post #27

vardump wrote: post #22

Earlier quoted context omitted.

> Which still means you have to write your code at least thrice, which is two times more than with a variable length SIMD ISA. 256 and 512 bits are the only reasonable widths. 256 bit AVX2 is what, 13 or 14 years old now.

no. Because Intel is full of absolute idiots, Intel atom didn't support AVX 1 until Gracemont. Tremont is missing AVX1, AVX2, FMA, and basically the rest of X86v3, and shipped in CPUs as recently as 2021 (Jasper Lake).

Intel also shipped a bunch of Pentium-branded CPUs that have AVX disabled, leading to oddities like a Kaby Lake based CPU that doesn't have AVX, and even worse, also shipped a few CPUs that have AVX2 but not BMI2:

https://sourceware.org/bugzilla/show_bug.cgi?id=29611

https://developercommunity.visualstudio.com/t/Crash-in-Windo...

Re: Fundamental flaws of SIMD ISAs (2021)

#66

Post by ack_complete » Fri, Apr 25, 2025, 1:50 AM UTC

PaulHoule wrote: post #28

TinkersW wrote: post #4

I write a lot of SIMD and I don't really agree with this.. Flaw1:fixed width I prefer fixed width as it makes the code simpler to write, size is known as compile time so we know the size of our structures. Swizzle algorithms are also customized based on the size. Flaw2:pipelining no CPU I care about is in order so mostly irrelevant, and even scalar instructions are pipelined Flaw3: tail handling I code with SIMD as t…

In AVX-512 we have a platform that rewards the assembly language programmer like few platforms have since the 6502. I see people doing really clever things that are specific to the system and one level it is really cool but on another level it means SIMD is the domain of the specialist, Intel puts out press releases about the really great features they have for the national labs and for Facebook whereas the rest of u…

AVX-512 also has a lot of wonderful facilities for autovectorization, but I suspect its initial downclocking effects plus getting yanked out of Alder Lake killed a lot of the momentum in improving compiler and library usage of it.

Even the Steam Hardware Survey, which is skewed toward upper end hardware, only shows 16% availability of baseline AVX-512, compared to 94% for AVX2.

Re: Fundamental flaws of SIMD ISAs (2021)

#67

Post by adgjlsfhk1 » Fri, Apr 25, 2025, 1:55 AM UTC

ack_complete wrote: post #66

PaulHoule wrote: post #28

Earlier quoted context omitted.

In AVX-512 we have a platform that rewards the assembly language programmer like few platforms have since the 6502. I see people doing really clever things that are specific to the system and one level it is really cool but on another level it means SIMD is the domain of the specialist, Intel puts out press releases about the really great features they have for the national labs and for Facebook whereas the rest of u…

AVX-512 also has a lot of wonderful facilities for autovectorization, but I suspect its initial downclocking effects plus getting yanked out of Alder Lake killed a lot of the momentum in improving compiler and library usage of it. Even the Steam Hardware Survey, which is skewed toward upper end hardware, only shows 16% availability of baseline AVX-512, compared to 94% for AVX2.

It will be interesting seeing what happens now that AMD is shipping good AVX-512. It really just makes Intel seem incompetent (especially since they're theoretically bringing AVX-512 back in next year anyway)

Re: Fundamental flaws of SIMD ISAs (2021)

#68

Post by dzaima » Fri, Apr 25, 2025, 1:58 AM UTC

dragontamer wrote: post #62

dzaima wrote: post #47

Earlier quoted context omitted.

Tail handling is not significant for loops with tons of iterations, but there are a ton of real-world situations where you might have a loop take only like 5 iterations or something (even at like 100 iterations, with a loop processing 8 elements at a time (i.e. 256-bit vectors, 32-bit elements), that's 12 vectorized iterations plus up to 7 scalar ones, which is still quite significant. At 1000 iterations you could st…

Note that AVX512 have per-lane execution masks so I'm not fully convinced that tail handling should even be a thing anymore. If(my lane is beyond the buffer) then (exec flag off, do not store my lane). Which in practice should be a simple vcompress instruction (AVX512 register) and maybe a move afterwards??? I admit that I'm not an AVX512 expert but it doesn't seem all that difficult with vcompress instructions + exe…

It takes like 4 instrs to compute the mask from an arbitrary length (AVX-512 doesn't have any instruction for this so you need to do `bzhi(-1, min(left,vl))` and move that to a mask register) so you still would likely want to avoid it in the hot loop.

Doing the tail separately but with masking SIMD is an improvement over a scalar loop perf-wise (..perhaps outside of the case of 1 or 2 elements, which is a realistic situation for a bunch of loops too), but it'll still add a double-digit percentage to code size over just a plain SIMD loop without tail handling.

And this doesn't help pre-AVX-512, and AVX-512 isn't particularly widespread (AVX2 does have masked load/store with 32-/64-bit granularity, but not 8-/16-bit, and the instrs that do exist are rather slow on AMD (e.g. unconditional 12 cycles/instr throughput for masked-storing 8 32-bit elements); SSE has none, and ARM NEON doesn't have any either (and ARM SVE isn't widespread either, incl. not supported on apple silicon))

(don't need vcompress, plain masked load/store instrs do exist in AVX-512 and are sufficient)

Re: Fundamental flaws of SIMD ISAs (2021)

#69

Post by dragontamer » Fri, Apr 25, 2025, 2:23 AM UTC

dzaima wrote: post #68

dragontamer wrote: post #62

Earlier quoted context omitted.

Note that AVX512 have per-lane execution masks so I'm not fully convinced that tail handling should even be a thing anymore. If(my lane is beyond the buffer) then (exec flag off, do not store my lane). Which in practice should be a simple vcompress instruction (AVX512 register) and maybe a move afterwards??? I admit that I'm not an AVX512 expert but it doesn't seem all that difficult with vcompress instructions + exe…

It takes like 4 instrs to compute the mask from an arbitrary length (AVX-512 doesn't have any instruction for this so you need to do `bzhi(-1, min(left,vl))` and move that to a mask register) so you still would likely want to avoid it in the hot loop. Doing the tail separately but with masking SIMD is an improvement over a scalar loop perf-wise (..perhaps outside of the case of 1 or 2 elements, which is a realistic s…

> It takes like 2 instrs to compute the mask from a length (AVX-512 doesn't have any instruction for this so you need to do a bzhi in GPR and move that to a mask register) so you still would likely want to avoid it in the hot loop.

Keep a register with the values IdxAdjustment = [0, 1, 2, 3, 4, 5, 6, 7].

ExecutionMask = (Broadcast(CurIdx) + IdxAdjustment) Keep looping while(any(vector) I'm not seeing this take up any "extra" instructions at all. You needed the while() loop after all. It costs +1 Vector Register (IdxAdjustment) and a kMask by my count.

> And this doesn't help pre-AVX-512, and AVX-512 isn't particularly widespread

AVX512 is over 10 years old now. And the premier SIMD execution instruction set is CUDA / NVidia, not AVX512.

AVX512 is now available on all AMD CPUs and has been for the last two generations. It is also available on a select number of Intel CPUs. There is also AMD RDNA, Intel Xe ISAs that could be targeted.

> instrs that do exist are rather slow on AMD (e.g. unconditional 12 cycles/instr throughput for masked-storing 8 32-bit elements);

Okay, I can see that possibly being an issue then.

EDIT: AMD Zen5 Optimization Manual states Latency1 and throughput 2-per-clocktick, while Intel's Skylake documentation of https://www.intel.com/content/www/us/en/docs/intrinsics-guid... states Latency5 Throughput 1-per-clock-tick.

AMD Zen5 seems to include support to vmovdqu8 (its in the optimization guide .xlsx sheet with latencies/throughputs, also as 1-latency / 4-throughput). This includes vmovdqu8 (

I'm not sure if the "mask" register changes the instruction. I'll do some research to see if I can verify your claim (I don't have my Zen5 computer built yet... but its soon).

Re: Fundamental flaws of SIMD ISAs (2021)

#70

Post by dzaima » Fri, Apr 25, 2025, 2:39 AM UTC

dragontamer wrote: post #69

dzaima wrote: post #68

Earlier quoted context omitted.

It takes like 4 instrs to compute the mask from an arbitrary length (AVX-512 doesn't have any instruction for this so you need to do `bzhi(-1, min(left,vl))` and move that to a mask register) so you still would likely want to avoid it in the hot loop. Doing the tail separately but with masking SIMD is an improvement over a scalar loop perf-wise (..perhaps outside of the case of 1 or 2 elements, which is a realistic s…

> It takes like 2 instrs to compute the mask from a length (AVX-512 doesn't have any instruction for this so you need to do a bzhi in GPR and move that to a mask register) so you still would likely want to avoid it in the hot loop. Keep a register with the values IdxAdjustment = [0, 1, 2, 3, 4, 5, 6, 7]. ExecutionMask = (Broadcast(CurIdx) + IdxAdjustment) Keep looping while(any(vector) I'm not seeing this take up any…

That's two instrs - bumping the indices, and doing the comparison. You still want scalar pointer/index bumping for contiguous loads/stores (using gathers/scatters for those would be stupid and slow), and that gets you the end check for free via fused cmp+jcc.

And those two instrs are vector instrs, i.e. competing with execution units for the actual thing you want to compute, whereas scalar instrs have at least some independent units that allow avoiding desiring infinite unrolling.

And if your loop is processing 32-bit (or, worse, smaller) elements, those indices, if done as 64-bit, as most code will do, will cost even more.

AVX-512 might be 10 years old, but Intel's latest (!) cores still don't support it on hardware with E-cores, so still a decade away from being able to just assume it exists. Another thread on this post mentioned that Intel has shipped hardware without AVX/AVX2/FMA as late as 2021 even.

> Okay, I can see that possibly being an issue then.

To be clear, that's only the AVX2 instrs; AVX-512 masked loads/stores are fast (..yes, even on Zen 4 where the AVX-512 masked loads/stores are fast, the AVX2 ones that do an equivalent amount of work (albeit taking the mask in a different register class) are slow). uops.info: https://uops.info/table.html?search=maskmovd%20m256&cb_lat=o...

Intel also has AVX-512 masked 512-bit 8-bit-elt stores at half the throughput of unmasked for some reason (not 256-bit or ≥16-bit-elt though; presumably culprit being the mask having 64 elts): https://uops.info/table.html?search=movdqu8%20m512&cb_lat=on...

And masked loads use some execution ports on both Intel and AMD, eating away from throughput of the main operation. All in all just not implemented for being able to needlessly use masked loads/stores in hot loops.