Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding SIMD: Infinite complexity of trivial problems

modular.com

Post reply on HN
Print view View original

111–120 of 127 posts

Re: Understanding SIMD: Infinite complexity of trivial problems

#111

Post by janwas »

dragontamer wrote: post #93
janwas wrote: post #75

Earlier quoted context omitted.

My understanding is that there is a lot of hand-writing (not just fine-tuning) going on. AFAIK CuDNN and TensorRT are written directly as SASS, not CUDA. And the presence of FP8 in H100, but not A100, would likely require a complete rewrite.

Cub, thrust and many other libraries that make those kernels possible don't need to be rewritten. When you write a merge sort in CUDA, you can keep it across all versions. Maybe the new instructions can improve a few corner cases, but it's not like AVX to AVX512 where you need to rewrite everything. Ex: https://github.com/NVIDIA/cub/blob/main/cub/device/device_me...

I agree not everything needs to be rewritten. And neither does code using an abstraction such as Highway, so we can stop beating that dead horse.

Re: Understanding SIMD: Infinite complexity of trivial problems

#112

Post by janwas »

kristianp wrote: post #109
janwas wrote: post #77

Earlier quoted context omitted.

Max performance is a stretch - recompilation would not utilize tensor cores, right? "too hard for mainstream programmers" seems overly pessimistic. I've run several workshops where devs have written dot-product kernels using Highway after 30 minutes of introduction.

They said intrinsics. Highway is an abstraction on top of intrinsics.

OK :) A thin abstraction, though. If comparing with alternative categories such as domain-specific language or autovectorization, I'd still classify Highway as intrinsics, just portable and easier to use.

Re: Understanding SIMD: Infinite complexity of trivial problems

#113

Post by janwas »

dragontamer wrote: post #95
janwas wrote: post #76

Earlier quoted context omitted.

I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions. The cost is that the binary carries around both AVX2 and AVX-512 codepaths, but that is not an issue IMO.

> I hope people aren't writing directly to AVX2. Did you not read the article? It's using AVX intrinsics and NEON intrinsics.

I did, and I truly do not understand why some people do this. As shown in the reddit comments on this article [1], the initial intrinsics version was quite suboptimal and clearly worse than portable code [2].

When not busy unnecessarily rewriting everything for each ISA, it is easier to see and have time for vital optimizations such as unrolling :)

[1]: https://www.reddit.com/r/cpp/comments/1gzob1g/understanding_... [2]: https://github.com/google/highway/blob/master/hwy/contrib/do...

Re: Understanding SIMD: Infinite complexity of trivial problems

#114

Post by janwas »

jandrewrogers wrote: post #101
janwas wrote: post #76

Earlier quoted context omitted.

I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions. The cost is that the binary carries around both AVX2 and AVX-512 codepaths, but that is not an issue IMO.

Many use cases for SIMD aren't trivially expressible through wrappers and abstractions. It is sometimes cleaner, easier, and produces more optimized codegen to write the intrinsics directly. It isn't ideal but it often produces the best result for the effort involved. An issue with the abstractions that does not go away is that the optimal code architecture -- well above the level of the SIMD wrappers -- is dependent…

Wrappers can be zero-overhead, so any claim of better codegen vs the underlying intrinsics sounds dubious. "best result for the [higher] effort involved" also contradicts my experience, so I ask for evidence.

One counterexample: our portable vqsort [1] outperforms AVX-512-specific intrinsics [2].

I agree that high-level design may differ. You seem aware that Highway, and probably also other wrappers, supports specializing code for some target(s), but possibly misunderstand how, given the "additional layer of indirection" claim. Wrappers give you a portable baseline, and remove some of the potholes and ugly syntax, but boil down to inlined wrapper functions.

If you want to specialize, that is supported. And what is the downside? Even if you say the benefit of a wrapper is reduced vs manually written intrinsics (and reinventing all the workarounds for their missing instructions), do you not agree that the benefit is still nonzero?

[1]: https://github.com/google/highway/tree/master/hwy/contrib/so... [2]: https://github.com/Voultapher/sort-research-rs/blob/38f37eef...

Re: Understanding SIMD: Infinite complexity of trivial problems

#115

Post by saagarjha »

Joker_vD wrote: post #5

> SIMD instructions are complex, and even Arm is starting to look more “CISCy” than x86! Thank you for saying it out loud. XLAT/XLATB of x86 is positively tame compared to e.g. vrgatherei16.vv/vrgather.vv.

That's RISC-V, no?

Re: Understanding SIMD: Infinite complexity of trivial problems

#116

Post by saagarjha »

benchmarkist wrote: post #38
fooblaster wrote: post #37

Earlier quoted context omitted.

It would probably be easier to clone the existing repository than get an llm to regurgitate llvm.

The AI would learn from llvm as well.

I think your comments would be improved if you learned from LLVM first.

Re: Understanding SIMD: Infinite complexity of trivial problems

#117

Post by saagarjha »

adrian_b wrote: post #78
dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

While there is some truth in what you say, it makes seem like writing in the CUDA style is something new and revolutionary invented by NVIDIA, which it is not. The CUDA style of writing parallel programs is nothing else than the use of the so-called "parrallel do" a.k.a. "parrallel for" program structure, which has been already discussed in 1963. Notable later evolutions of this concept have been present in "Communic…

I'm not sure what you mean. All CUDA code needs to be aware of the programming model the GPU imposes on them, splitting their code manually into threads and warps and blocks and kernels to match. This isn't really transparent at all.

Re: Understanding SIMD: Infinite complexity of trivial problems

#118

Post by saagarjha »

janwas wrote: post #114
jandrewrogers wrote: post #101

Earlier quoted context omitted.

Many use cases for SIMD aren't trivially expressible through wrappers and abstractions. It is sometimes cleaner, easier, and produces more optimized codegen to write the intrinsics directly. It isn't ideal but it often produces the best result for the effort involved. An issue with the abstractions that does not go away is that the optimal code architecture -- well above the level of the SIMD wrappers -- is dependent…

Wrappers can be zero-overhead, so any claim of better codegen vs the underlying intrinsics sounds dubious. "best result for the [higher] effort involved" also contradicts my experience, so I ask for evidence. One counterexample: our portable vqsort [1] outperforms AVX-512-specific intrinsics [2]. I agree that high-level design may differ. You seem aware that Highway, and probably also other wrappers, supports special…

The downside is that you write an implementation in Highway, find that it doesn't perform how you want, and then you have to rewrite it.

Re: Understanding SIMD: Infinite complexity of trivial problems

#119

Post by saagarjha »

dragontamer wrote: post #19
dist-epoch wrote: post #10

Earlier quoted context omitted.

> If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs That's not true. For maximum performance you need to tweak the code to a particular GPU model/architecture. Intel has SSE/AVX/AVX2/AVX512, but CUDA has like 10 iterations of this (increasing capabilities). Code written 15 years ago would not use modern capabilities, like more flexible m…

Maximum performance? Okay, you'll have to upgrade to ballot instructions or whatever and rearchitect your algorithms. (Or other wavefront / voting / etc. etc. new instructions that have been invented. Especially those 4x4 matrix multiplication AI instructions). But CUDA -> PTX intermediate code has allowed for significantly more flexibility. For crying out loud, the entire machine code (aka SASS) of NVidia GPUs has b…

This is not really fair or true. Nvidia changes the meaning of PTX when they want to. For example, warp thread divergence is something they implemented in an architecture revision, technically breaking existing code. With SM90 (Hopper) they have even started including unstable features in PTX that they reduce promises for even further. And of course everyone who cares about performance is rewriting their kernels (or using someone else's rewritten kernels) for each new architecture. I honestly do not think it is fair to compare this to the CPU landscape, which has much stronger backwards compatibility guarantees.

Re: Understanding SIMD: Infinite complexity of trivial problems

#120

Post by janwas »

saagarjha wrote: post #118
janwas wrote: post #114

Earlier quoted context omitted.

Wrappers can be zero-overhead, so any claim of better codegen vs the underlying intrinsics sounds dubious. "best result for the [higher] effort involved" also contradicts my experience, so I ask for evidence. One counterexample: our portable vqsort [1] outperforms AVX-512-specific intrinsics [2]. I agree that high-level design may differ. You seem aware that Highway, and probably also other wrappers, supports special…

The downside is that you write an implementation in Highway, find that it doesn't perform how you want, and then you have to rewrite it.

Curious - how is/was performance helped by rewriting? Why not reach out to us, to see if it can be fixed in the library - wouldn't that be cheaper than rewriting?
Post reply on HN