Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fundamental flaws of SIMD ISAs (2021)

bitsnbites.eu

Post reply on HN
Print view View original

51–60 of 146 posts

Re: Fundamental flaws of SIMD ISAs (2021)

#51

Post by Remnant44 »

dzaima wrote: post #42
Remnant44 wrote: post #38

Earlier quoted context omitted.

These days, I strongly believe that loop unrolling is a pessimization, especially with SIMD code. Scalar code should be unrolled by the compiler to the SIMD word width to expose potential parallelism. But other than that, correctly predicted branches are free, and so is loop instruction overhead on modern wide-dispatch processors. For example, even running a maximally efficient AVX512 kernel on a zen5 machine that di…

Intel still shares ports between vector and scalar on P-cores; a scalar multiply in the loop will definitely fight with a vector port, and the bits of pointer bumps and branch and whatnot can fill up the 1 or 2 scalar-only ports. And maybe there are some minor power savings from wasting resources on the scalar overhead. Still, clang does unroll way too much.

My understanding is that they've changed this for Lion Cove and all future P cores, moving to much more of a Zen-like setup with seperate schedulers and ports for vector and scalar ops.

Re: Fundamental flaws of SIMD ISAs (2021)

#52

Post by PaulHoule »

bee_rider wrote: post #50
PaulHoule wrote: post #28

Earlier quoted context omitted.

In AVX-512 we have a platform that rewards the assembly language programmer like few platforms have since the 6502. I see people doing really clever things that are specific to the system and one level it is really cool but on another level it means SIMD is the domain of the specialist, Intel puts out press releases about the really great features they have for the national labs and for Facebook whereas the rest of u…

It is kind of a bummer that MKL isn’t open sourced, as that would make inclusion in Linux easier. It is already free-as-in-beer, but of course that doesn’t solve everything. Baffling that MS didn’t use it. They have a pretty close relationship… Agree that they are sort of going after hard-to-use niche features nowadays. But I think it is just that the real thing we want—single threaded performance for branchy code—is…

At the very least you can decode UTF-8 really quickly with AVX-512

https://lemire.me/blog/2023/08/12/transcoding-utf-8-strings-...

and web browsers at the very least spent a lot of cycles on decoding HTML and Javascript which is UTF-8 encoded. It turns out AVX-512 is good at a lot of things you wouldn't think SIMD would be good at. Intel's got the problem that people don't want to buy new computers because they don't see much benefit from buying a new computer, but a new computer doesn't have the benefit it could have because of lagging software support, and the software support lags because there aren't enough new computers to justify the work to do the software support. Intel deserves blame for a few things, one of which is that they have dragged their feet at getting really innovative features into their products while turning people off with various empty slogans.

They really do have a new instruction set that targets plain ordinary single threaded branchy code

https://www.intel.com/content/www/us/en/developer/articles/t...

they'll probably be out of business before you can use it.

Re: Fundamental flaws of SIMD ISAs (2021)

#53

Post by jonstewart »

jcranmer wrote: post #17
camel-cdr wrote: post #12

Earlier quoted context omitted.

> I prefer fixed width Do you have examples for problems that are easier to solve in fixed-width SIMD? I maintain that most problems can be solved in a vector-length-agnostic manner. Even if it's slightly more tricky, it's certainly easier than restructuring all of your memory allocations to add padding and implementing three versions for all the differently sized SIMD extensions your target may support. And you can…

There's a category of autovectorization known as Superword-Level Parallelism (SLP) which effectively scavenges an entire basic block for individual instruction sequences that might be squeezed together into a SIMD instruction. This kind of vectorization doesn't work well with vector-length-agnostic ISAs, because you generally can't scavenge more than a few elements anyways, and inducing any sort of dynamic vector len…

A number of the cool string processing SIMD techniques depend a _lot_ on register widths and instruction performance characteristics. There’s a fair argument to be made that x64 could be made more consistent/legible for these use cases, but this isn’t matmul—whether you have 128, 256, or 512 bits matters hugely and you may want entirely different algorithms that are contingent on this.

Re: Fundamental flaws of SIMD ISAs (2021)

#54

Post by winwang »

LegionMammal978 wrote: post #9
pornel wrote: post #6

There are alternative universes where these wouldn't be a problem. For example, if we didn't settle on executing compiled machine code exactly as-is, and had a instruction-updating pass (less involved than a full VM byte code compilation), then we could adjust SIMD width for existing binaries instead of waiting decades for a new baseline or multiversioning faff. Another interesting alternative is SIMT. Instead of hav…

> Another interesting alternative is SIMT. Instead of having a handful of special-case instructions combined with heavyweight software-switched threads, we could have had every instruction SIMDified. It requires structuring programs differently, but getting max performance out of current CPUs already requires SIMD + multicore + predictable branching, so we're doing it anyway, just in a roundabout way. Is that not whe…

Fun fact which I'm 50%(?) sure of: a single branch divergence for integer instructions on current nvidia GPUs won't hurt perf, because there are only 16 int32 lanes anyway.

Re: Fundamental flaws of SIMD ISAs (2021)

#55

Post by __abadams__ »

kevingadd wrote: post #34
cjbgkagh wrote: post #7

Earlier quoted context omitted.

I have similar thoughts, I don't understand the push for variable width SIMD. Possibly due to ignorance but I think it's an abstraction that can be specialized for different hardware so the similar tradeoffs between low level languages and high level languages apply. Since I already have to be aware of hardware level concepts such as 256bit shuffle not working across 128bit lanes and different instructions having ver…

Part of the motive behind variable width SIMD in WASM is that there's intentionally-ish no mechanism to do feature detection at runtime in WASM. The whole module has to be valid on your target, you can't include a handful of invalid functions and conditionally execute them if the target supports 256-wide or 512-wide SIMD. If you want to adapt you have to ship entire modules for each set of supported feature flags and…

It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this. Clang does it well if you use its built-in vector types (which can be any length). Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you just statically picked a size, even if it's not the native size.

Re: Fundamental flaws of SIMD ISAs (2021)

#56

Post by jandrewrogers »

derf_ wrote: post #49
TinkersW wrote: post #4

I write a lot of SIMD and I don't really agree with this.. Flaw1:fixed width I prefer fixed width as it makes the code simpler to write, size is known as compile time so we know the size of our structures. Swizzle algorithms are also customized based on the size. Flaw2:pipelining no CPU I care about is in order so mostly irrelevant, and even scalar instructions are pipelined Flaw3: tail handling I code with SIMD as t…

> I code with SIMD as the target, and have special containers that pad memory to SIMD width... I think this may be domain-specific. I help maintain several open-source audio libraries, and wind up being the one to review the patches when people contribute SIMD for some specific ISA, and I think without exception they always get the tail handling wrong. Due to other interactions it cannot always be avoided by padding.…

> Masked loads/stores are an improvement, but not universally available.

Traditionally we’ve worked around this with pretty idiomatic hacks that efficiently implement “masked load” functionality in SIMD ISAs that don’t have them. We could probably be better about not making people write this themselves every time.

Re: Fundamental flaws of SIMD ISAs (2021)

#57

Post by dzaima »

Remnant44 wrote: post #51
dzaima wrote: post #42

Earlier quoted context omitted.

Intel still shares ports between vector and scalar on P-cores; a scalar multiply in the loop will definitely fight with a vector port, and the bits of pointer bumps and branch and whatnot can fill up the 1 or 2 scalar-only ports. And maybe there are some minor power savings from wasting resources on the scalar overhead. Still, clang does unroll way too much.

My understanding is that they've changed this for Lion Cove and all future P cores, moving to much more of a Zen-like setup with seperate schedulers and ports for vector and scalar ops.

Oh, true, mistook it for an E-core while clicking through diagrams due to the port spam.. Still, that's a 2024 microarchirecture, it'll be like a decade before it's reasonable to ignore older ones.

Re: Fundamental flaws of SIMD ISAs (2021)

#58

Post by jandrewrogers »

__abadams__ wrote: post #55
kevingadd wrote: post #34

Earlier quoted context omitted.

Part of the motive behind variable width SIMD in WASM is that there's intentionally-ish no mechanism to do feature detection at runtime in WASM. The whole module has to be valid on your target, you can't include a handful of invalid functions and conditionally execute them if the target supports 256-wide or 512-wide SIMD. If you want to adapt you have to ship entire modules for each set of supported feature flags and…

It would be very easy to support 512-bit vectors everywhere, and just emulate them on most systems with a small number of smaller vectors. It's easy for a compiler to generate good code for this. Clang does it well if you use its built-in vector types (which can be any length). Variable-length vectors, on the other hand, are a very challenging problem for compiler devs. You tend to get worse code out than if you just…

The risk of 512-bit vectors everywhere is that many algorithms will spill the registers pretty badly if implemented in e.g. 128-bit vectors under the hood. In such cases you may be better off with a completely different algorithm implementation.

Re: Fundamental flaws of SIMD ISAs (2021)

#59

Post by jandrewrogers »

camel-cdr wrote: post #29
jandrewrogers wrote: post #18

Earlier quoted context omitted.

I also prefer fixed width. At least in C++, all of the padding, alignment, etc is automagically codegen-ed for the register type in my use cases, so the overhead is approximately zero. All the complexity and cost is in specializing for the capabilities of the underlying SIMD ISA, not the width. The benefit of fixed width is that optimal data structure and algorithm design on various microarchitectures is dependent on…

> The argument for vector width agnostic code is seems predicated on the proverbial “sufficiently advanced compiler”. A SIMD ISA having a fixed size or not is orthogonal to autovectorization. E.g. I've seen a bunch of cases where things get autovectorized for RVV but not for AVX512. The reason isn't fixed vs variable, but rather the supported instructions themselves. There are two things I'd like from a "sufficiently…

The interplay of SIMD width and microarchitecture is more important for performance engineering than you seem to be assuming. Those codegen decisions are made at layer above anything being talked about here and they operate on explicit awareness of things like register size.

It isn’t “same instruction but wider or narrower” or anything that can be trivially autovectorized, it is “different algorithm design”. Compilers are not yet rewriting data structures and algorithms based on microarchitecture.

I write a lot of SIMD code, mostly for database engines, little of which is trivial “processing a vector of data types” style code. AVX512 in particular is strong enough of an ISA that it is used in all kinds of contexts that we traditionally wouldn’t think of as a good for SIMD. You can build all kinds of neat quasi-scalar idioms with it and people do.

Re: Fundamental flaws of SIMD ISAs (2021)

#60

Post by bobmcnamara »

Remnant44 wrote: post #38
sweetjuly wrote: post #10

Loop unrolling isn't really done because of pipelining but rather to amortize the cost of looping. Any modern out-of-order core will (on the happy path) schedule the operations identically whether you did one copy per loop or four. The only difference is the number of branches.

These days, I strongly believe that loop unrolling is a pessimization, especially with SIMD code. Scalar code should be unrolled by the compiler to the SIMD word width to expose potential parallelism. But other than that, correctly predicted branches are free, and so is loop instruction overhead on modern wide-dispatch processors. For example, even running a maximally efficient AVX512 kernel on a zen5 machine that di…

> The cost of unrolling is decreased code density and reduced effectiveness of the instruction / uOp cache.

There are some cases where useful code density goes up.

Ex: unroll the Goertzel algorithm by a even number, and suddenly the entire delay line overhead evaporates.

Post reply on HN