Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AVX2 is slower than SSE2-4.x under Windows ARM emulation

blogs.remobjects.com

Post reply on HN
Print view View original

51–60 of 88 posts

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#51

Post by mtklein »

If I remember correctly, the AVX2 feature set is a fairly direct upscale of SSE4.1 to 256 bit. Very few instructions even allowed interaction between the top and bottom 128 bits, I assume to make implementation on existing 128 bit vector units easier. And the most notable new things that AVX2 added beyond that widening, fp16 conversion and FMA support, are also present in NEON, so I wouldn't expect that to be the issue either.

So I'd bet the issue is either newness of the codebase, as the article suggests, or perhaps that it is harder to schedule the work in 256 bit chunks than 128. It's got to be easier when you've got more than enough NEON q registers to handle the xmms, harder when you've got only exactly enough to pair up for handling ymms?

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#52

Post by jsheard »

camel-cdr wrote: post #49
jsheard wrote: post #46

Earlier quoted context omitted.

> You can treat both SVE and RVV as a regular fixed-width SIMD ISA. Kind of, but the part which looks particularly annoying is that you can't put variable-width vectors on the stack or pass them around as values in most languages, because they aren't equipped to handle types with unknown size at compile time. ARM seems to be proposing a C language extension which does require compilers to support variably sized types…

> Kind of, but the part which looks particularly annoying is that you can't put variable-width vectors on the stack or pass them around as values in most languages, because they aren't equipped to handle types with unknown size at compile time Yes, you can't, which is annoying, but you can if you compile for a specific vector length. This is mostly a library structure problem. E.g. simdjson has a generic backend that…

> Toolchain support should also be improved, e.g. you could make all vregs take 512-bit on the stack, but have the codegen only utilize the lowee 128 bit, if you have 128-but vregs, 256-bit if you have 256-bit vregs and 512-bit if you have >=512-bit vregs.

SVE theoretically supports hardware up to 2048-bit, so conservatively reserving the worst-case size at compile time would be pretty wasteful. That's 16x overhead in the base case of 128-bit hardware.

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#53

Post by happyPersonR »

Aurornis wrote: post #47
kbolino wrote: post #5

I suspected this was because the vector units were not wide enough, and it seems that is the case. AVX2 is 256-bit, ARM NEON is only 128-bit. The big question then is, why are ARM desktop (and server?) cores so far behind on wider SIMD support? It's not like Intel/AMD came up with these extensions for x86 yesterday; AVX2 is over 15 years old.

> The big question then is, why are ARM desktop (and server?) cores so far behind on wider SIMD support? Very wide SIMD instructions require a lot of die space and a lot of power. The AVX-512 implementation in Intel's Knight's Landing took up 40% of the die area (Source https://chipsandcheese.com/p/knights-landing-atom-with-avx-5... which is an excellent site for architectural analysis) Most ARM desktop/mobile parts…

Yeah this seems likely, but with all the LLM stuff it might be an outdated assumption.

Buy new chips next year! Haha :)

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#54

Post by ack_complete »

mtklein wrote: post #51

If I remember correctly, the AVX2 feature set is a fairly direct upscale of SSE4.1 to 256 bit. Very few instructions even allowed interaction between the top and bottom 128 bits, I assume to make implementation on existing 128 bit vector units easier. And the most notable new things that AVX2 added beyond that widening, fp16 conversion and FMA support, are also present in NEON, so I wouldn't expect that to be the iss…

There are also mode switching and calling convention issues.

The way that the vector registers were extended to 256-bit causes problems when legacy 128-bit and 256-bit ops are mixed. Doing so puts the CPU into a mode where all legacy 128-bit ops are forced to blend the high half, which can reduce throughput of existing SSE2-based library routines to as low as 1/4 throughput. For this reason, AVX code has to aggressively use the VZEROUPPER instruction to ensure that the CPU is not left in AVX 256-bit vector mode before possibly returning to any library or external code that uses SSE2. VZEROUPPER sets a flag to zero the high half of all 256-bit registers, so it's cheap on modern x86 CPUs but can be expensive to emulate without hardware support.

The other problem is that only the low 128 bits of vector registers are preserved across function calls due to the Windows x64 calling convention and the VZEROUPPER issue. This means that practically any call to external code forces the compiler to spill all AVX vectors to memory. Ideally 256-bit vector usage is concentrated in leaf routines so this isn't an issue, but where used in non-leaf routines, it can result in a lot of memory traffic.

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#55

Post by spacecadet_ »

mtklein wrote: post #51

If I remember correctly, the AVX2 feature set is a fairly direct upscale of SSE4.1 to 256 bit. Very few instructions even allowed interaction between the top and bottom 128 bits, I assume to make implementation on existing 128 bit vector units easier. And the most notable new things that AVX2 added beyond that widening, fp16 conversion and FMA support, are also present in NEON, so I wouldn't expect that to be the iss…

> Very few instructions even allowed interaction between the top and bottom 128 bits

That would be plain AVX, AVX2 has shuffles across the 128-bit boundary. To me that seems like the main hurdle for emulation with 128-bit vectors, in my experience compilers are very eager to emit shuffle instructions if allowed, and emulating a 256-bit shuffle with 128-bit operations would require 2 shuffles and a blend for each half of the emulated register.

EDIT: I just noticed that the benchmark in the article is pure math which probably wouldn't hit this particular issue, so this doesn't explain the performance difference...

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#56

Post by corysama »

jorvi wrote: post #12
SecretDreams wrote: post #3

Earlier quoted context omitted.

Almost every x86 cpu made in the last decade should have avx2. Maybe you're thinking of avx512 or avx10?

Yeah, sounds like they're confusing AVX2 for AVX512. AVX2 has been common for a decade at least and greatly accelerates performance. AVX512 is so kludgy that it usually leads to a detriment in performance due to the extreme power requirements triggering thermal throttling.

In gamedev it takes 7-10 years before you can require a new tech without getting a major backlash. AMD came out with AVX2 support in 2015. And, the (vocal minority) petitions to get AVX2 requirements removed from major games and VR systems are only now starting to quiet down.

So, in order to make use of users new fancy hardware without abandoning other users old and busted hardware, you have to support multiple back-ends. Same as it ever was.

Actually, a lot easier than it ever was today. Doom 3 famously required Carmack to reimplement the rendering 6 times to get the same results out of 6 different styles of GPUs that were popular at the time.

ARB Basic Fallback (R100) Multi-pass Minimal effects, no specular.

NV10 GeForce 2 / 4 MX, 5 Passes, Used Register Combiners.

NV20 GeForce 3 / 4 Ti, 2–3 Passes, Vertex programs + Combiners.

R200 Radeon 8500–9200, 1 Pass, Used ATI_fragment_shader.

NV30 GeForce FX Series, 1 Pass, Precision optimizations (FP16).

ARB2 Radeon 9500+ / GF 6+, 1 Pass, Standard high-end GLSL-like assembly.

https://community.khronos.org/t/doom-3/37313

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#57

Post by cogman10 »

TheJoeMan wrote: post #31

I tried searching "SSE2-4.x" and this is the top result in DDG and Google, so I was initially confused what instruction set the article is referring to. However, this appears to be shorthand for SSE2 through SSE4? Perhaps a rephrasing of the article title could be helpful.

I read it as SSE2->4.x.

Generally speaking, when working with SSE instructions you'll end up using a mix of instructions from 2->4 as they are all effectively just additional operations on the SSE2 registers.

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#58

Post by vintagedave »

bhouston wrote: post #19
kbolino wrote: post #5

I suspected this was because the vector units were not wide enough, and it seems that is the case. AVX2 is 256-bit, ARM NEON is only 128-bit. The big question then is, why are ARM desktop (and server?) cores so far behind on wider SIMD support? It's not like Intel/AMD came up with these extensions for x86 yesterday; AVX2 is over 15 years old.

Hasn't there been issues with AVX2 causing such a heavy load on the CPU that frequency scaling would kick in a lot of cases slowing down the whole CPU? https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#Dow... My experience is that trying to get benefits from the vector extensions is incredibly hard and the use cases are very narrow. Having them in a standard BLAS implementation, sure, but outside of that I thi…

Re hard to get benefits: a lot depends on the compiler. In Elements (the toolchain this article was tested with) we made a bunch of modifications to LLVM passes to prioritise vectorisation in situations where it could, but did not.

I've heard anecdotally that the old pre-LLVM Intel C++ Compiler also focused heavily on vectorisation and had some specific tradeoffs to achieve it. I think they use LLVM now too and for all I know they've made similar modifications that we did. But we see a decent number of code patterns that can and now are optimised.

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#59

Post by cogman10 »

iberator wrote: post #2

AVX2 should be banned anyway. Only expensive CPUs have it, ruining mininum games requirements and making hardware obsolete. Most of the world lives of 300$ per month

Any x86 CPU manufactured in the last 10 year has AVX2.

Here's a laptop for $350 which has a CPU with AVX2 support.

https://ebay.us/m/yoznZ1

Re: AVX2 is slower than SSE2-4.x under Windows ARM emulation

#60

Post by jovial_cavalier »

josefx wrote: post #26
jovial_cavalier wrote: post #13

Earlier quoted context omitted.

A ton of vector math applications these days are high dimensional vector spaces. A good example of that for arm would I guess be something like fingerprint or face id. Also, it doesn't just speed up vector math. Compilers these days with knowledge of these extensions can auto-vectorize your code, so it has the potential to speed up every for-loop you write.

> A good example of that for arm would I guess be something like fingerprint or face id. So operations that are not performance critical and are needed once or twice every hour? Are you sure you don't want to include a dedicated cluster of RTX 6090 Ti GPUs to speed them up?

I'd argue that those are actually very performance critical because if it takes 5 seconds to unlock your phone, you're going to get a new phone.

The point is taken, though, that seemingly the performance is fine as it is for these applications. My point was only that you don't need to be running state of the art LLMs to be using vector math with more than 4 dimensions.

Post reply on HN