Related:
Three Fundamental Flaws of SIMD - https://news.ycombinator.com/item?id=28114934 - Aug 2021 (20 comments)
Fundamental flaws of SIMD ISAs (2021)
71–80 of 146 posts
Re: Fundamental flaws of SIMD ISAs (2021)
#72Earlier quoted context omitted.
> It takes like 2 instrs to compute the mask from a length (AVX-512 doesn't have any instruction for this so you need to do a bzhi in GPR and move that to a mask register) so you still would likely want to avoid it in the hot loop. Keep a register with the values IdxAdjustment = [0, 1, 2, 3, 4, 5, 6, 7]. ExecutionMask = (Broadcast(CurIdx) + IdxAdjustment) Keep looping while(any(vector) I'm not seeing this take up any…
That's two instrs - bumping the indices, and doing the comparison. You still want scalar pointer/index bumping for contiguous loads/stores (using gathers/scatters for those would be stupid and slow), and that gets you the end check for free via fused cmp+jcc. And those two instrs are vector instrs, i.e. competing with execution units for the actual thing you want to compute, whereas scalar instrs have at least some i…
Gotcha. Makes sense. Thanks for the discussion!
Overall, I agree that AVX and Neon have their warts and performance issues. But they're like 15+ years old now and designed well before GPU Compute was possible.
> using gathers/scatters for those would be stupid and slow
This is where CPUs are really bad. GPUs will coalesce gather/scatters thanks to __shared__ memory (with human assistance of course).
But also the simplest of load/store patters are auto-detected and coalesced. So a GPU programmer doesn't have to worry about SIMD lane load/store (called vgather in AVX512) being slower. It's all optimized to hell and back.
Having a full lane-to-lane crossbar and supporting high performance memory access patterns needs to be a priority moving forward.
Re: Fundamental flaws of SIMD ISAs (2021)
#73Earlier quoted context omitted.
AVX-512 also has a lot of wonderful facilities for autovectorization, but I suspect its initial downclocking effects plus getting yanked out of Alder Lake killed a lot of the momentum in improving compiler and library usage of it. Even the Steam Hardware Survey, which is skewed toward upper end hardware, only shows 16% availability of baseline AVX-512, compared to 94% for AVX2.
It will be interesting seeing what happens now that AMD is shipping good AVX-512. It really just makes Intel seem incompetent (especially since they're theoretically bringing AVX-512 back in next year anyway)
No proof, but I suspect that AMD's AVX-512 support played a part in Intel dumping AVX10/256 and changing plans back to shipping a full 512-bit consumer implementation again (we'll see when they actually ship it).
The downside is that AMD also increased the latency of all formerly cheap integer vector ops. This removes one of the main advantages against NEON, which historically has had richer operations but worse latencies. That's one thing I hope Intel doesn't follow.
Also interesting is that Intel's E-core architecture is improving dramatically compared to the P-core, even surpassing it in some cases. For instance, Skymont finally has no penalty for denormals, a long standing Intel weakness. Would not be surprising to see the E-core architecture take over at some point.
Re: Fundamental flaws of SIMD ISAs (2021)
#74Personally, I think load and increment address register in a single instruction is extremely valuable here. It's not quite the risc model but I think that it is actually pretty significant in avoiding a von nurmon bottleneck with simd (the irony in this statement)
I found that a lot of the custom simd cores I've written for simply cannot issue instructions fast enough risvc. Or when they it's in quick bursts and than increments and loop controls that leave the engine idling for more than you'd like.
Better dual issue helps but when you have seperate vector queue you are sending things to it's not that much to add increments into vloads and vstores
Re: Fundamental flaws of SIMD ISAs (2021)
#75I write a lot of SIMD and I don't really agree with this.. Flaw1:fixed width I prefer fixed width as it makes the code simpler to write, size is known as compile time so we know the size of our structures. Swizzle algorithms are also customized based on the size. Flaw2:pipelining no CPU I care about is in order so mostly irrelevant, and even scalar instructions are pipelined Flaw3: tail handling I code with SIMD as t…
I agree; and the article seems to have also quite a few technical flaws: - Register width: we somewhat maxed out at 512 bits, with Intel going back to 256 bits for non-server CPUs. I don't see larger widths on the horizon (even if SVE theoretically supports up to 2048 bits, I don't know any implementation with ~~>256~~ >512 bits). Larger bit widths are not beneficial for most applications and the few applications tha…
I think the variable length stuff does solve encoding issues, and RISCV takes so big strides with the ideas around chaining and vl/lmul/vtype registers.
I think they would benefit from having 4 vtype registers, though. It's wasted scalar space, but how often do you actually rotate between 4 different vector types in main loop bodies. The answer is pretty rarely. And you'd greatly reduce the swapping between vtypes when. I think they needed to find 1 more bit but it's tough the encoding space isn't that large for rvv which is a perk for sure
Can't wait to seem more implementions of rvv to actually test some of it's ideas
Re: Fundamental flaws of SIMD ISAs (2021)
#76I write a lot of SIMD and I don't really agree with this.. Flaw1:fixed width I prefer fixed width as it makes the code simpler to write, size is known as compile time so we know the size of our structures. Swizzle algorithms are also customized based on the size. Flaw2:pipelining no CPU I care about is in order so mostly irrelevant, and even scalar instructions are pipelined Flaw3: tail handling I code with SIMD as t…
> I code with SIMD as the target, and have special containers that pad memory to SIMD width... I think this may be domain-specific. I help maintain several open-source audio libraries, and wind up being the one to review the patches when people contribute SIMD for some specific ISA, and I think without exception they always get the tail handling wrong. Due to other interactions it cannot always be avoided by padding.…
It depends on how integrated your SIMD strategy is into the overall technical design. Tail handling is much easier if you can afford SIMD-friendly padding so a full vector load/store is possible even if you have to manually mask. That avoids a lot of the hassle of breaking down memory accesses just to avoid a page fault or setting off the memory checker.
Beyond that -- unit testing. I don't see enough of it for vectorized routines. SIMD widths are small enough that you can usually just test all possible offsets right up against a guard page and brute force verify that no overruns occur.
Re: Fundamental flaws of SIMD ISAs (2021)
#77Earlier quoted context omitted.
The tricky part with reductions is that they are somewhat inherently slow since they often need to be done pairwise and a pairwise reduction over 16 elements will naturally have pretty limited parallelism.
kinda? this is sort of a direct result of the 'vectors are just sliced registers' model. if i do a pairwise operation and divide my domain by 2 at each step, is the resulting vector sparse or dense? if its dense then I only really top out when i'm in the last log2slice steps.
Yes, but this is not cheap for hardware. CPU designers love SIMD because it lets them just slap down ALUs in parallel and get 32x performance boosts. Reductions, however, are not entirely parallel and instead have a relatively huge gate depth.
For example, suppose an add operation has a latency of one unit in some silicon process. To add reduce a 32 element vector, you'll have a five deep tree, which means your operation has a latency of five units. You can pipeline this, but you can't solve the fact that this operation has a 5x higher latency than the non-reduce operations.
Re: Fundamental flaws of SIMD ISAs (2021)
#78Earlier quoted context omitted.
I agree; and the article seems to have also quite a few technical flaws: - Register width: we somewhat maxed out at 512 bits, with Intel going back to 256 bits for non-server CPUs. I don't see larger widths on the horizon (even if SVE theoretically supports up to 2048 bits, I don't know any implementation with ~~>256~~ >512 bits). Larger bit widths are not beneficial for most applications and the few applications tha…
I think the variable length stuff does solve encoding issues, and RISCV takes so big strides with the ideas around chaining and vl/lmul/vtype registers. I think they would benefit from having 4 vtype registers, though. It's wasted scalar space, but how often do you actually rotate between 4 different vector types in main loop bodies. The answer is pretty rarely. And you'd greatly reduce the swapping between vtypes wh…
If you had two extra bits in the instruction encoding, I think it'd make much more sense to encode element width directly in instructions, leaving LMUL multiplier & agnosticness settings in vsetvl; only things that'd suffer then would be if you need tail-undisturbed for one instr (don't think that's particularly common) and fancy things that reinterpret the vector between different element widths (very uncommon).
Will be interesting to see if longer encodings for RVV with encoded vtype or whatever ever materialize.
Re: Fundamental flaws of SIMD ISAs (2021)
#79Earlier quoted context omitted.
Low power RISC cores (both ARM and RISC-V) are typically in-order actually! But any core I can think of as 'high-performance' is OOO.
MIPS as well as Alpha AFAIR. And technically itanium, otoh It seems to me a bit like a niche for any performance advantages...
I would not call neither MIPS, Alpha nor Itanium "high-performance" in 2025...
Re: Fundamental flaws of SIMD ISAs (2021)
#80There are alternative universes where these wouldn't be a problem. For example, if we didn't settle on executing compiled machine code exactly as-is, and had a instruction-updating pass (less involved than a full VM byte code compilation), then we could adjust SIMD width for existing binaries instead of waiting decades for a new baseline or multiversioning faff. Another interesting alternative is SIMT. Instead of hav…
> There are alternative universes where these wouldn't be a problem
Do people that say these things have literally any experience of merit?
> For example, if we didn't settle on executing compiled machine code exactly as-is, and had a instruction-updating pass
You do understand that at the end of the day, hardware is hard (fixed) and software is soft (malleable) right? There will be always be friction at some boundary - it doesn't matter where you hide the rigidity of a literal rock, you eventually reach a point where you cannot reconfigure something that you would like to. And also the parts of that rock that are useful are extremely expensive (so no one is adding instruction-updating pass silicon just because it would be convenient). That's just physics - the rock is very small but fully baked.
> we could have had every instruction SIMDified
Tell me you don't program GPUs without telling me. Not only is SIMT a literal lie today (cf warp level primitives), there is absolutely no reason to SIMDify all instructions (and you better be a wise user of your scalar registers and scalar instructions if you want fast GPU code).
I wish people would just realize there's no grand paradigm shift that's coming that will save them from the difficult work of actually learning how the device works in order to be able to use it efficiently.