Isn’t k-means memory bandwidth bound? What was the arithmetic intensity of the final code?
I assume that the image would at least fit in L3.
AVX-512: First Impressions on Performance and Programmability
41–50 of 54 posts
Re: AVX-512: First Impressions on Performance and Programmability
#42Earlier quoted context omitted.
Also trying to let the compilers know that the float* are aligned would be a good move. auto aligned_p = std::assume_aligned (p)
which honestly, shouldn't be neccessary today with avx512. There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput, while if it_is_ aligned you will get the same performance as the aligned-only load. No reason for the compiler to balk at vectorizing unaligned data these days.
Even with AVX512, memory arguments used in most instructions (those that are not explicitly unaligned loads) need to be aligned, no? E.g., for vaddps zmm0, zmm0, [rdi] (saving a register and an instruction over vmovups + vaddps reg, reg, reg), rdi must be suitably aligned.
Apart from that, there indeed hasn't been a real unaligned (non-atomic) penalty on Intel since Nehalem or something. Although there definitely is an extra cost for crossing a page, and I would assume also a smaller one for crossing a cache line—which is quite relevant when your ops are the same size as one!
Re: AVX-512: First Impressions on Performance and Programmability
#43> CUDA architects [...] happily exposed every ugly details of underlying hardware to the programmer if that allows a bit more performance. After spending more than a decade dancing around all the underlying x86 hidden stuff for low-level optimization, I appreciate CUDA a lot. Everything is there under your total control. No more one size fits all. Higher barrier of entry but no surprises and less time spent debugging…
Then you’re locked into the ecosystem and whims of signed proprietary drivers so in a way you have no control whatsoever.
Sure. But Intel's beancounter board is way more scary [1] and moving from CUDA to AMD's ROCm isn't that hard, anyway.
[1] "Intel Officially Introduces Pay-As-You-Go Chip Licensing - Intel's Xeon Sapphire Rapids CPUs to activate additional features on demand" https://www.tomshardware.com/news/intel-officially-introduce... (2022)
Re: AVX-512: First Impressions on Performance and Programmability
#44A few gentle points: (a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of. (b) You show a problem wher…
> (d) AVX512 is quite hardware specific. Some processors execute these instructions much better than others. It's really a collection of extensions, and you get frequency downclocking that's better/worse on different processors as these instructions are executed. To re-iterate, this is our observation as well. The first AVX512 processors would execute such code quite fast for a short time, then overheat and throttle,…
There is a large variance in AVX-512 performance across the microarchitectures, particularly the early ones. In addition to throttling, which was mitigated relatively early, some relatively feature-complete microarchitectures (e.g. Ice Lake) were sensitive to alignment. The microarchitectures with these issues are approaching obsolescence at this point. AVX-512 runs very well with predictable performance on most vaguely recent microarchitectures.
In my case I find AVX-512 to be usable in practice because the kinds of users that are obsessed with performance-engineered systems are typically not running creaky hardware.
Re: AVX-512: First Impressions on Performance and Programmability
#45Earlier quoted context omitted.
which honestly, shouldn't be neccessary today with avx512. There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput, while if it_is_ aligned you will get the same performance as the aligned-only load. No reason for the compiler to balk at vectorizing unaligned data these days.
Even with AVX512, memory arguments used in most instructions (those that are not explicitly unaligned loads) need to be aligned, no? E.g., for vaddps zmm0, zmm0, [rdi] (saving a register and an instruction over vmovups + vaddps reg, reg, reg), rdi must be suitably aligned. Apart from that, there indeed hasn't been a real unaligned (non-atomic) penalty on Intel since Nehalem or something. Although there definitely is…
With the older microarchitectures there was a large penalty for crossing a cache line with AVX-512. In some cases, the performance could be worse than AVX2!
In older microarchitectures like Ice Lake it was pretty bad, so you wanted to avoid unaligned loads if you could. This penalty has rapidly shrunk across subsequent generations of microarchitectures. The penalty is still there but on recent microarchitectures it is small enough that the unaligned case often isn't a showstopper.
The main reason to use aligned loads in code is to denote cases where you expect the address to always be aligned i.e. it should blow up if it isn't. Forcing alignment still makes sense if you want predictable, maximum performance but it isn't strictly necessary for good performance on recent hardware in the way it used to be.
Re: AVX-512: First Impressions on Performance and Programmability
#46A few gentle points: (a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of. (b) You show a problem wher…
Hi, thanks for reading.
Re (b) I'm curious what that middle ground is. Is there any simple refactor to help GCC to get rid of this `if`? (Note, ISPC did fine here)
(c) Just to be clear, all the codes in benchmark figures (baseline and SIMD) were compiled with fast-math flags.
Regarding (a), one of the points I wanted to get across was that it didn't feel that complicated to program in the end as I had thought. Porting to AVX-512 felt mechanical (hence the success of LLMs in one-shotting the whole thing).
This is a subjective opinion, depends on programmer's experience etc- so I won't dwell on it. I just wish more CPU programmers gave it a try.
Re: AVX-512: First Impressions on Performance and Programmability
#47Isn’t k-means memory bandwidth bound? What was the arithmetic intensity of the final code?
No. Assuming `k` is small enough, which in practice often is, the arithmetic intensity of this kernel is 25-90 Flops/Byte, way above the roofline knee of any modern CPU.
Re: AVX-512: First Impressions on Performance and Programmability
#48I found this a weird article. If you wish to see some speedups using AVX512, without limiting yourself to C or C++, you might want to try ISPC ( https://ispc.github.io/index.html ). You'll get sane aliasing rules from the perspective of performance, multi-target binaries with dynamic dispatching and a lot more control over the code generated.
Hi, I actually mentioned ISPC several times there. And although I strenuously avoided crowning one approach "better" over the other, it is worth pointing out that 1) Many of these benefits of ISPC can be had from explicit SIMD libraries like Google's Highway, and 2) ISPC (or any SIMT model) is a departure from how the underlying hardware works, and as the AI community is discovering with GPU, this abstraction can sometimes be lot more headache than its worth.
Re: AVX-512: First Impressions on Performance and Programmability
#49Earlier quoted context omitted.
which honestly, shouldn't be neccessary today with avx512. There's essentially no reason to prefer the aligned load/store commands over the unaligned ones - if the actual pointer is unaligned it will function correctly at half the throughput, while if it_is_ aligned you will get the same performance as the aligned-only load. No reason for the compiler to balk at vectorizing unaligned data these days.
Even with AVX512, memory arguments used in most instructions (those that are not explicitly unaligned loads) need to be aligned, no? E.g., for vaddps zmm0, zmm0, [rdi] (saving a register and an instruction over vmovups + vaddps reg, reg, reg), rdi must be suitably aligned. Apart from that, there indeed hasn't been a real unaligned (non-atomic) penalty on Intel since Nehalem or something. Although there definitely is…
AVX doesn't require alignment of any memory operands, with the exception of the specific load aligned instruction. So you/the compiler are free to use the reg,mem form interchangibly with unaligned data.
The penalty on modern machines is an extra cycle of latency and, when crossing a cacheline, half the throughput (AVX512 always crosses a cacheline since they are cacheline sized!). These are pretty mild penalties given what you gain! So while it's true that peak L1 cache performance is gained when everything is aligned.. the blocker is elsewhere for most real code.
Re: AVX-512: First Impressions on Performance and Programmability
#50A few gentle points: (a) You mention that the NVidia docs push people to use libraries, etc. to really get high performance CUDA kernels, rather than writing them themselves. My argument would be that SIMD is exactly the same - they're something really that are perfect if you're writing a BLAS implementation but are too low level for most developers thinking about a problem to make use of. (b) You show a problem wher…
Hi, thanks for reading. Re (b) I'm curious what that middle ground is. Is there any simple refactor to help GCC to get rid of this `if`? (Note, ISPC did fine here) (c) Just to be clear, all the codes in benchmark figures (baseline and SIMD) were compiled with fast-math flags. Regarding (a), one of the points I wanted to get across was that it didn't feel that complicated to program in the end as I had thought. Portin…
Fort what it's worth, I had the exact same experience you did when I started writing SIMD code explicitly with intrinsics.
I avoided it for a long time because, well, it was so damn ugly and verbose to do simple things. However, in actual practice it's not nearly as painful as it looks, and you get used to it quickly.