Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Understanding SIMD: Infinite complexity of trivial problems

modular.com

Post reply on HN
Print view View original

21–30 of 127 posts

Re: Understanding SIMD: Infinite complexity of trivial problems

#21

Post by variadix »

dragontamer wrote: post #6

Intel needs to see what has happened to their AVX instructions and why NVidia has taken over. If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs rather than being forced to write and rewrite in SSE vs AVX vs AVX512. GPU SIMD is still SIMD. Just... better at it. I think AMD and Intel GPUs can keep up btw. But software advantage and long te…

How much of this is because CUDA is designed for GPU execution and because the GPU ISA isn’t a stable interface? E.g. new GPU instructions can be utilized by new CUDA compilers for new hardware because the code wasn’t written to a specific ISA? Also, don’t people fine tune GPU kernels per architecture manually (either by hand or via automated optimizers that test combinations in the configuration space)?

Re: Understanding SIMD: Infinite complexity of trivial problems

#22

Post by dist-epoch »

dragontamer wrote: post #19
dist-epoch wrote: post #10

Earlier quoted context omitted.

> If you just wrote your SIMD in CUDA 15 years ago, NVidia compilers would have given you maximum performance across all NVidia GPUs That's not true. For maximum performance you need to tweak the code to a particular GPU model/architecture. Intel has SSE/AVX/AVX2/AVX512, but CUDA has like 10 iterations of this (increasing capabilities). Code written 15 years ago would not use modern capabilities, like more flexible m…

Maximum performance? Okay, you'll have to upgrade to ballot instructions or whatever and rearchitect your algorithms. (Or other wavefront / voting / etc. etc. new instructions that have been invented. Especially those 4x4 matrix multiplication AI instructions). But CUDA -> PTX intermediate code has allowed for significantly more flexibility. For crying out loud, the entire machine code (aka SASS) of NVidia GPUs has b…

Intel code from 15 years ago also runs today. But it will not use AVX512.

Which is the same with PTX, right? If you didn't use the tensor core instructions or wavefront voting in the CUDA code, the PTX generated from it will not either, and NVIDIA will not magically add those capabilities in when compiling to SASS.

Maybe it remains competitive because the code is inherently parallel anyway, so it will naturally scale to fill the extra execution units of the GPU, which is where most of the improvement is generation to generation.

While AVX code can't automatically scale to use the AVX512 units.

Re: Understanding SIMD: Infinite complexity of trivial problems

#23

Post by bob1029 »

I see a lot of "just use the GPU" and you'd often be right.

SIMD on the CPU is most compelling to me due to the latency characteristics. You are nanoseconds away from the control flow. If the GPU needs some updated state regarding the outside world, it takes significantly longer to propagate this information.

For most use cases, the GPU will win the trade off. But, there is a reason you don't hear much about systems like order matching engines using them.

Re: Understanding SIMD: Infinite complexity of trivial problems

#25

Post by ashvardanian »

EVa5I7bHFq9mnYK wrote: post #12

C# vectors do a great job of simplifying those intrinsics in a safe and portable manner.

There are dozens of libraries, frameworks, and compiler toolchains that try to abstract away SIMD capabilities, but I don't think it's a great approach.

The only 2 approaches that still make sense to me:

A. Writing serial vectorization-aware code in a native compiled language, hoping your compiler will auto-vectorize.

B. Implementing natively for every hardware platform, as the ISA differences are too big to efficiently abstract away anything beyond 128-register float multiplication and addition.

This article, in a way, an attempt to show how big the differences even for simple data-parallel floating-point tasks.

Re: Understanding SIMD: Infinite complexity of trivial problems

#26

Post by pclmulqdq »

bob1029 wrote: post #23

I see a lot of "just use the GPU" and you'd often be right. SIMD on the CPU is most compelling to me due to the latency characteristics. You are nanoseconds away from the control flow. If the GPU needs some updated state regarding the outside world, it takes significantly longer to propagate this information. For most use cases, the GPU will win the trade off. But, there is a reason you don't hear much about systems…

You would be surprised. The GPU often loses even for small neural nets given the large latency. Anything that needs high throughput or is sized like an HPC problem should use a GPU, but a lot of code benefits from SIMD on small problems.

Re: Understanding SIMD: Infinite complexity of trivial problems

#27

Post by Conscat »

juancn wrote: post #13

The main problem is that there are no good abstractions in popular programming languages to take advantage of SIMD extensions. Also, the feature set being all over the place (e.g. integer support is fairly recent) doesn't help either. ISPC is a good idea, but execution is meh... it's hard to setup and integrate. Ideally you would want to be able to easily use this from other popular languages, like Java, Python, Java…

I think the EVE library for C++ is a great abstraction. It's got an unusual syntax using subscript operator overloading, but that winds up being a very ergonomic and flexible way to program with masked-SIMD.

Re: Understanding SIMD: Infinite complexity of trivial problems

#28

Post by [deleted] »

bob1029 wrote: post #23

I see a lot of "just use the GPU" and you'd often be right. SIMD on the CPU is most compelling to me due to the latency characteristics. You are nanoseconds away from the control flow. If the GPU needs some updated state regarding the outside world, it takes significantly longer to propagate this information. For most use cases, the GPU will win the trade off. But, there is a reason you don't hear much about systems…

[deleted]

Re: Understanding SIMD: Infinite complexity of trivial problems

#30

Post by dzaima »

ashvardanian wrote: post #25
EVa5I7bHFq9mnYK wrote: post #12

C# vectors do a great job of simplifying those intrinsics in a safe and portable manner.

There are dozens of libraries, frameworks, and compiler toolchains that try to abstract away SIMD capabilities, but I don't think it's a great approach. The only 2 approaches that still make sense to me: A. Writing serial vectorization-aware code in a native compiled language, hoping your compiler will auto-vectorize. B. Implementing natively for every hardware platform, as the ISA differences are too big to efficien…

There's the middle-ground approach of having primarily target-specific operations but with intersecting ones named the same, and allowing easily building custom abstractions on top of such to paper over the differences how best it makes sense for the given application. That's the approach https://github.com/mlochbaum/Singeli takes.

There's a good amount of stuff that can clearly utilize SIMD without much platform-specificness, but doesn't easily autovectorize - early-exit checks in a loop, packed bit boolean stuff, some data rearranging, probing hashmap checks, some very-short-variable-length-loop things. And while there might often be some parts that do just need to be entirely target-specific, they'll usually be surrounded by stuff that doesn't (the loop, trip count calculation, loads/stores, probably some arithmetic).

Post reply on HN