Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding SIMD: Infinite complexity of trivial problems

modular.com

Post reply on HN
Print view View original

71–80 of 127 posts

Re: Understanding SIMD: Infinite complexity of trivial problems

#71

Post by Earw0rm »

ashvardanian wrote: post #25
EVa5I7bHFq9mnYK wrote: post #12

C# vectors do a great job of simplifying those intrinsics in a safe and portable manner.

There are dozens of libraries, frameworks, and compiler toolchains that try to abstract away SIMD capabilities, but I don't think it's a great approach. The only 2 approaches that still make sense to me: A. Writing serial vectorization-aware code in a native compiled language, hoping your compiler will auto-vectorize. B. Implementing natively for every hardware platform, as the ISA differences are too big to efficien…

The library approach does a pretty good job in conjunction with a good compiler, and sensible algorithm design.

You're writing C++ code but as if it was shader code.

I've seen impressive results with clang doing this sort of thing.

Re: Understanding SIMD: Infinite complexity of trivial problems

#72

Post by Earw0rm »

juancn wrote: post #13

The main problem is that there are no good abstractions in popular programming languages to take advantage of SIMD extensions. Also, the feature set being all over the place (e.g. integer support is fairly recent) doesn't help either. ISPC is a good idea, but execution is meh... it's hard to setup and integrate. Ideally you would want to be able to easily use this from other popular languages, like Java, Python, Java…

std::experimental::simd is happening. It should be part of c++26.

Re: Understanding SIMD: Infinite complexity of trivial problems

#73

Post by Earw0rm »

moldavi wrote: post #33
bob1029 wrote: post #23

I see a lot of "just use the GPU" and you'd often be right. SIMD on the CPU is most compelling to me due to the latency characteristics. You are nanoseconds away from the control flow. If the GPU needs some updated state regarding the outside world, it takes significantly longer to propagate this information. For most use cases, the GPU will win the trade off. But, there is a reason you don't hear much about systems…

Do Apple's chips (M1 etc) change this at all, since they share memory with the GPU?

Not much. Synchronisation of tasks is still a big overhead.

If you've got tens to hundreds of microseconds worth of workload, sure, get the GPU to do it.

But bear in mind 1000 clocks at 4GHz is 250ns, there's still a sizeable region where tight CPU/GPU integration isn't tight enough.

Re: Understanding SIMD: Infinite complexity of trivial problems

#74

Post by janwas »

jabl wrote: post #69
dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

Sometimes I wonder about an alternative history scenario where CPU ISA's would have chosen a SIMT style model instead of SIMD. "Just" have something like fork/join instructions to start/stop vector mode, otherwise use the standard scalar instructions in both scalar and vector mode. Would have avoided a lot of combinatorial explosion in instructions. (of course you'd have to do something for cross-lane operations, and…

Not sure why SIMT would help, it requires more compiler transforms than if the code is written for packets/vectors or whatever we want to call them. As you note, cross-lane is a key part of a good SIMD abstraction. Vulkan calls it "subgroups", but from where I sit it's still SIMD.

Re: Understanding SIMD: Infinite complexity of trivial problems

#75

Post by janwas »

variadix wrote: post #21
dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

How much of this is because CUDA is designed for GPU execution and because the GPU ISA isn’t a stable interface? E.g. new GPU instructions can be utilized by new CUDA compilers for new hardware because the code wasn’t written to a specific ISA? Also, don’t people fine tune GPU kernels per architecture manually (either by hand or via automated optimizers that test combinations in the configuration space)?

My understanding is that there is a lot of hand-writing (not just fine-tuning) going on. AFAIK CuDNN and TensorRT are written directly as SASS, not CUDA. And the presence of FP8 in H100, but not A100, would likely require a complete rewrite.

Re: Understanding SIMD: Infinite complexity of trivial problems

#76

Post by janwas »

dragontamer wrote: post #34
dist-epoch wrote: post #22

Earlier quoted context omitted.

Intel code from 15 years ago also runs today. But it will not use AVX512. Which is the same with PTX, right? If you didn't use the tensor core instructions or wavefront voting in the CUDA code, the PTX generated from it will not either, and NVIDIA will not magically add those capabilities in when compiling to SASS. Maybe it remains competitive because the code is inherently parallel anyway, so it will naturally scale…

It's not the same. AVX2 instructions haven't changed and never will change. In contrast, NVidia can go from 64-bit instruction bundles to 128-bit machine code (96-bit instruction + 32-bit control information) between Pascal (aka PTX Compute Capacity 5) and Voltage (aka PTX Compute Capacity 7) and all the old PTX code just autocompiles to the new assembly instruction format and takes advantage of all the new memory ba…

I hope people aren't writing directly to AVX2. When using a wrapper such as Highway, you get exactly this kind of update after a recompile, or even just running your code on a CPU that supports newer instructions.

The cost is that the binary carries around both AVX2 and AVX-512 codepaths, but that is not an issue IMO.

Re: Understanding SIMD: Infinite complexity of trivial problems

#77

Post by janwas »

dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

Max performance is a stretch - recompilation would not utilize tensor cores, right?

"too hard for mainstream programmers" seems overly pessimistic. I've run several workshops where devs have written dot-product kernels using Highway after 30 minutes of introduction.

Re: Understanding SIMD: Infinite complexity of trivial problems

#78

Post by adrian_b »

dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

While there is some truth in what you say, it makes seem like writing in the CUDA style is something new and revolutionary invented by NVIDIA, which it is not.

The CUDA style of writing parallel programs is nothing else than the use of the so-called "parrallel do" a.k.a. "parrallel for" program structure, which has been already discussed in 1963. Notable later evolutions of this concept have been present in "Communicating Sequential Processes" by C.A.R. Hoare (1978-08: "arrays of processes"), then in the programming language Occam, which was designed based on what Hoare had described, then in the OpenMP extension of Fortran (1997-10), then in the OpenMP extension of C and C++ (1998-10).

Programming in CUDA does not bring anything new, except that in comparison e.g. with OpenMP some keywords are implicit and others are different, so the equivalence is not immediately obvious.

Programming for CPUs in the much older OpenMP is equivalent with programming in CUDA for GPUs.

The real innovation of NVIDIA has been the high quality of the NVIDIA CUDA compiler and CUDA runtime GPU driver, which are able to distribute the work that must be done on the elements of an array over all the available cores, threads and SIMD lanes, in a manner that is transparent for the programmer, so in many cases the programmer is free to ignore which is the actual structure of the GPU that will run the program.

Previous compilers for OpenMP or for other such programming language extensions for parallel programming have been much less capable to produce efficient parallel programs without being tuned by the programmer for each hardware variant.

Re: Understanding SIMD: Infinite complexity of trivial problems

#79

Post by exDM69 »

ashvardanian wrote: post #25
EVa5I7bHFq9mnYK wrote: post #12

C# vectors do a great job of simplifying those intrinsics in a safe and portable manner.

There are dozens of libraries, frameworks, and compiler toolchains that try to abstract away SIMD capabilities, but I don't think it's a great approach. The only 2 approaches that still make sense to me: A. Writing serial vectorization-aware code in a native compiled language, hoping your compiler will auto-vectorize. B. Implementing natively for every hardware platform, as the ISA differences are too big to efficien…

I will respectfully disagree with your statement, with the caveat that I mostly dabble in arithmetic with 128b/256b float and int vectors.

Using C or C++ with vector extensions (Gcc/Clang) or Rust (nightly) std::simd is very easy and you get code that is portable to different CPUs and ISAs.

But most importantly they have a zero cost fallback option to CPU-specific intrinsics when you need them. An f32x8 can be passed at zero cost as __mm256 to any core::arch::x86_64::__mm_intrinsic (or xmmintrin.h in C++ land).

You gain portable arithmetic and swizzles and SIMD vector types, but lose nothing. Not having to write everything for x86_64 and aarch64 is a huge win even if doesn't quite cover everything.

Additionally you can use wider vectors than your hardware supports, the compiler is able to split your f64x64 to 128, 256 or 512 bit registers as needed depending on the compile target.

Re: Understanding SIMD: Infinite complexity of trivial problems

#80

Post by janwas »

Earw0rm wrote: post #72
juancn wrote: post #13

The main problem is that there are no good abstractions in popular programming languages to take advantage of SIMD extensions. Also, the feature set being all over the place (e.g. integer support is fairly recent) doesn't help either. ISPC is a good idea, but execution is meh... it's hard to setup and integrate. Ideally you would want to be able to easily use this from other popular languages, like Java, Python, Java…

std::experimental::simd is happening. It should be part of c++26.

Unfortunately a bit late :) Highway reached v1.0 about 2.5 years ago. How long would it take until Clang/GCC/MSVC are ready, and all users' distros have updated? Not to mention that the number of ops provided by std::experimental::simd is extremely limited - basically only math operators, and zero support for shuffling/crypto/rounding/interleaving/table lookups which seem indispensable for many applications.
Post reply on HN