Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding SIMD: Infinite complexity of trivial problems

modular.com

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Understanding SIMD: Infinite complexity of trivial problems

#101

Post by jandrewrogers »

janwas wrote: post #76
dragontamer wrote: post #34

Earlier quoted context omitted.

It's not the same. AVX2 instructions haven't changed and never will change. In contrast, NVidia can go from 64-bit instruction bundles to 128-bit machine code (96-bit instruction + 32-bit control information) between Pascal (aka PTX Compute Capacity 5) and Voltage (aka PTX Compute Capacity 7) and all the old PTX code just autocompiles to the new assembly instruction format and takes advantage of all the new memory ba…

I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions. The cost is that the binary carries around both AVX2 and AVX-512 codepaths, but that is not an issue IMO.

Many use cases for SIMD aren't trivially expressible through wrappers and abstractions. It is sometimes cleaner, easier, and produces more optimized codegen to write the intrinsics directly. It isn't ideal but it often produces the best result for the effort involved.

An issue with the abstractions that does not go away is that the optimal code architecture -- well above the level of the SIMD wrappers -- is dependent on the capabilities of the silicon. The wrappers can't solve for that. And if you optimize the code architecture for the silicon architecture, it quickly approximates writing architecture-specific intrinsics with an additional layer of indirection, which significantly reduces any notional benefit from the abstractions.

The wrappers can't abstract enough, and higher level abstractions (written with architecture aware intrinsics) are often too use case specific to reuse widely.

Re: Understanding SIMD: Infinite complexity of trivial problems

#102

Post by mmoskal »

ashvardanian wrote: post #57
dragontamer wrote: post #34

Earlier quoted context omitted.

It's not the same. AVX2 instructions haven't changed and never will change. In contrast, NVidia can go from 64-bit instruction bundles to 128-bit machine code (96-bit instruction + 32-bit control information) between Pascal (aka PTX Compute Capacity 5) and Voltage (aka PTX Compute Capacity 7) and all the old PTX code just autocompiles to the new assembly instruction format and takes advantage of all the new memory ba…

There is still a lot of similarity between CPU and GPU programming - between AVX and PTX. Different generations of CPU cores handle the same AVX2 instructions differently. The microcode changes and the schedulers change, but the process is transparent for the user, similar to PTX.

I imagine there is and order of magnitude of difference between how much you can translate in software, with large memory and significant time budget to work with, compared to microcode.

Re: Understanding SIMD: Infinite complexity of trivial problems

#103

Post by Remnant44 »

janwas wrote: post #76
dragontamer wrote: post #34

Earlier quoted context omitted.

It's not the same. AVX2 instructions haven't changed and never will change. In contrast, NVidia can go from 64-bit instruction bundles to 128-bit machine code (96-bit instruction + 32-bit control information) between Pascal (aka PTX Compute Capacity 5) and Voltage (aka PTX Compute Capacity 7) and all the old PTX code just autocompiles to the new assembly instruction format and takes advantage of all the new memory ba…

I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions. The cost is that the binary carries around both AVX2 and AVX-512 codepaths, but that is not an issue IMO.

I would wager that most real world SIMD use is with direct intrinsics.

Re: Understanding SIMD: Infinite complexity of trivial problems

#105

Post by synack »

dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

I’ve been playing with a new Lunar Lake laptop and they’ve complicated things even further with the Neural Processing Unit (NPU)

Now if your vectors are INT8/FP8 you’re supposed to shovel them into this accelerator via PCIe, rather than packing into registers for AVX512.

I wish they’d just pick an interface for vector ops and stick with it.

Re: Understanding SIMD: Infinite complexity of trivial problems

#106

Post by colejohnson66 »

juancn wrote: post #13

The main problem is that there are no good abstractions in popular programming languages to take advantage of SIMD extensions. Also, the feature set being all over the place (e.g. integer support is fairly recent) doesn't help either. ISPC is a good idea, but execution is meh... it's hard to setup and integrate. Ideally you would want to be able to easily use this from other popular languages, like Java, Python, Java…

C#’s Vector does a pretty great job.

Re: Understanding SIMD: Infinite complexity of trivial problems

#107

Post by dragontamer »

mmoskal wrote: post #102
ashvardanian wrote: post #57

Earlier quoted context omitted.

There is still a lot of similarity between CPU and GPU programming - between AVX and PTX. Different generations of CPU cores handle the same AVX2 instructions differently. The microcode changes and the schedulers change, but the process is transparent for the user, similar to PTX.

I imagine there is and order of magnitude of difference between how much you can translate in software, with large memory and significant time budget to work with, compared to microcode.

Most CPU instructions are 1-to-1 with their microcode. I dare say that microcode is nearly irrelevant, any high-performance instruction (ex: multiply, add, XOR, etc. etc.) is but a single instruction anyway.

Load/Store are memory dependent in all architectures. So that's just a different story as CPUs and GPUs have completely different ideas of how caches should work. (CPUs aim for latency, GPUs for bandwidth + incredibly large register spaces with substantial hiding of latency thanks to large occupancies).

-------------

That being said: reorder buffers on CPUs are well over 400-instructions these days, with super-large cores (like Apple's M4) is apparently on the order of 600 to 800 instructions.

Reorder buffers are _NOT_ translation. They're Tomasulo's algorithm (https://en.wikipedia.org/wiki/Tomasulo%27s_algorithm). If you want to know how CPUs do out-of-order, study that.

I'd say CPUs have small register spaces (16 architectural registers, maybe 32), but large register files of maybe 300 or 400+. Tomasulo's algorithm is used to out-of-order access registers.

You should think of instructions like "mov rax, [memory]" as closer to "rax = malloc(register); delayed-load(rax, memory); Out-of-order execute all instructions that don't use RAX ahead of us in instruction stream".

Tomasulo's algorithm means using ~300-register file to _pretend_ to be just 16 architectural registers. The 300 registers keeps the data out-of-order and allows you to execute. Registers in modern CPUs are closer to unique_ptr in C++, assigning them frees (aka: reorder buffer) and also mallocs a new register off the register-file.

Re: Understanding SIMD: Infinite complexity of trivial problems

#108

Post by david-gpu »

dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

> If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512

NVidia compilers would have compiled your code into something functional, but if you want to approach peak performance you need to at least tweak your kernels, and sometimes rewrite them from scratch. See for example the various MMA instructions that were introduced over time.

Edit: I see somebody made a similar comment and you addressed it. Sorry for the churn.

Re: Understanding SIMD: Infinite complexity of trivial problems

#109

Post by kristianp »

janwas wrote: post #77
dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

Max performance is a stretch - recompilation would not utilize tensor cores, right? "too hard for mainstream programmers" seems overly pessimistic. I've run several workshops where devs have written dot-product kernels using Highway after 30 minutes of introduction.

They said intrinsics. Highway is an abstraction on top of intrinsics.

Re: Understanding SIMD: Infinite complexity of trivial problems

#110

Post by kristianp »

> Let's explore these challenges and how Mojo helps address them

You've not linked to or explained what Mojo is. There's also a lot going on with different products mentioned: Modular, Unum cloud, SimSIMD that are not contextualised either. While I'm at it, where do the others come in (Ovadia, Lemire, Lattner), you all worked on SimSIMD, I guess?

That said, this is a great article, thanks.

Edit: Mojo is a programming language with python-like syntax, and is a product by Modular: https://github.com/modularml/mojo

Post reply on HN