Towards fearless SIMD • Hacker News Board

Re: Towards fearless SIMD

#41

Post by atq2119 » Wed, Oct 24, 2018, 7:21 PM UTC

m_mueller wrote: post #20

ndesaulniers wrote: post #16

> Code written for too high a SIMD capability will generally crash (it’s undefined behavior), while code written for too low a SIMD capability will fall short of the performance available. I can't help but think that SIMD intrinsics are the wrong level of abstraction. Literally assembly being moved into a high level language but without the ability for the compiler to optimize well. Hand written intrinsics can really…

As someone coming from CUDA I think you‘re on the right path. CUDA unifies multicore and vectorization parallelims. Crucial here is hardware support for branching (including early returns) even if it degrades performance. This allows a CUDA kernel being programmed in a programmer friendly way that is often already fairly performant, and the actual mapping to hardware is deferred to kernel invocation where it‘s straig…

I agree that this space needs to be explored more. ISPC is basically this, and it had some great ideas, it just needs to be more broadly available and integrated into build systems.

Re: Towards fearless SIMD

#42

Post by gameswithgo » Wed, Oct 24, 2018, 7:43 PM UTC

grandinj wrote: post #36

stochastic_monk wrote: post #14

I’ve found template metaprogramming to be an efficient, generic way to generate SIMD-accelerated code. (See [0].) By giving the type functions associated with each operation, the same interface can support 128, 256, or 512-bit SIMD, depending on hardware. It’s C++, so while it’s outside of the Rust ecosystem, it’s still a workable solution I’ve used in a half dozen projects. [0] https://github.com/dnbaker/vec

That doesn't give you run-time dispatch based on available hardware capabilities. So you either need virtual functions (and take the associated perf hit), or some other solution like the linker tricks mentions, or cached JIT, or....

it can, you generate all desired version of simd-ified function at compile time, then write a function to select the right one at runtime.

that is what the Rust libs are doing, just using traits or macros instead of templates.

Re: Towards fearless SIMD

#43

Post by marmaduke » Wed, Oct 24, 2018, 7:45 PM UTC

Const-me wrote: post #30

nikic wrote: post #27

Earlier quoted context omitted.

As usual, things aren't quite that simple. While it certainly can happen that the compiler can pessimize carefully crafted SIMD code by applying undesirable transformations, simply not optimizing intrinsics would lead to other problems. If you consider SIMD code that spans more than just one small function, then it is quite useful if the compiler can SROA (e.g. you pass around an array of vectors instead of having 8…

> it is quite useful if the compiler can SROA __attribute__((always_inline)) / __forceinline usually help. > LICM (you're calling a function that needs a constant vector in a loop) I can calculate that vector outside of the loop. > link in an assembly file. Way more complex, I need to write outer scalar code in assembly too, need to manually allocate registers, also C++ templates are sometimes very useful for that ki…

Can’t just isolate those in a compilation unit with -O0?

Re: Towards fearless SIMD

#44

Post by marmaduke » Wed, Oct 24, 2018, 7:48 PM UTC

m_mueller wrote: post #20

ndesaulniers wrote: post #16

> Code written for too high a SIMD capability will generally crash (it’s undefined behavior), while code written for too low a SIMD capability will fall short of the performance available. I can't help but think that SIMD intrinsics are the wrong level of abstraction. Literally assembly being moved into a high level language but without the ability for the compiler to optimize well. Hand written intrinsics can really…

As someone coming from CUDA I think you‘re on the right path. CUDA unifies multicore and vectorization parallelims. Crucial here is hardware support for branching (including early returns) even if it degrades performance. This allows a CUDA kernel being programmed in a programmer friendly way that is often already fairly performant, and the actual mapping to hardware is deferred to kernel invocation where it‘s straig…

OpenCL on CPU is exactly this and does very well. Last real world test I did, an 8 core Haswell and a GTX 970 were similar in performance, for the kernel I was running. Caveats apply of course but it was refreshing to have a single programming model.

Re: Towards fearless SIMD

#45

Post by joshvm » Wed, Oct 24, 2018, 7:49 PM UTC

m_mueller wrote: post #20

ndesaulniers wrote: post #16

> Code written for too high a SIMD capability will generally crash (it’s undefined behavior), while code written for too low a SIMD capability will fall short of the performance available. I can't help but think that SIMD intrinsics are the wrong level of abstraction. Literally assembly being moved into a high level language but without the ability for the compiler to optimize well. Hand written intrinsics can really…

As someone coming from CUDA I think you‘re on the right path. CUDA unifies multicore and vectorization parallelims. Crucial here is hardware support for branching (including early returns) even if it degrades performance. This allows a CUDA kernel being programmed in a programmer friendly way that is often already fairly performant, and the actual mapping to hardware is deferred to kernel invocation where it‘s straig…

Would a better example be OpenCL (at least the vision of it)? OpenCL kernels can run on both CPU and GPU, and unlike CUDA, it's not vendor locked. Though I've only really used CUDA, I don't know how comparable OpenCL is these days.

Re: Towards fearless SIMD

#46

Post by shepmaster » Wed, Oct 24, 2018, 7:53 PM UTC

neogodless wrote: post #4

lossolo wrote: post #3

Earlier quoted context omitted.

Single Instruction Multiple Data https://en.wikipedia.org/wiki/SIMD ELI5: Your processor can process more data in parallel. If you have for example loop that has something like this in the body: a[i] + b[i] = c[i] Using SIMD processor can make all this operations same time: a[i] + b[i] = c[i] a[i+1] + b[i+1] = c[i+1] a[i+2] + b[i+2] = c[i+2] a[i+3] + b[i+3] = c[i+3] Normally in one iteration you would only get one of…

Thanks! I get that if you don't already know the acronym, you're not the audience, but it is helpful to not have to search externally for a reference. Could you imagine if that was in a general programming magazine before global internet search engines were popular? You'd have to cross your fingers that your local encyclopedia has an entry! ETA: My bad - I did not think to click that. Embarrassing. Thanks for pointin…

What is "ETA"? /s :-)

Re: Towards fearless SIMD

#47

Post by brian-armstrong » Wed, Oct 24, 2018, 8:08 PM UTC

ndesaulniers wrote: post #16

> Code written for too high a SIMD capability will generally crash (it’s undefined behavior), while code written for too low a SIMD capability will fall short of the performance available. I can't help but think that SIMD intrinsics are the wrong level of abstraction. Literally assembly being moved into a high level language but without the ability for the compiler to optimize well. Hand written intrinsics can really…

Clang supports vector types for C and they're still nowhere near as fast as using intrinsics. For whatever various reasons, compilers tend to do pretty poorly at producing properly vectorized code. I think part of this is that it's hard to encode all the information you need to vectorize properly (this loop will run some power of 2 number of times, these pointers won't alias, etc)

Even using intrinsics is dicey sometimes - compilers spill registers more often than can make sense. I think the ideal situation would be if you could somehow hint at register scheduling without writing asm yourself, but compiler are still a long ways from that.

Re: Towards fearless SIMD

#48

Post by brian-armstrong » Wed, Oct 24, 2018, 8:11 PM UTC

wyldfire wrote: post #6

The simplest target-independent fearless SIMD is autovectorization[1] . But taking maximum advantage of that probably means writing some code that feels a little unnatural. Also, IIRC bounds checks thwart some autovectorization. [1] https://llvm.org/docs/Vectorizers.html

Autovectorizing is quite a long way away from being as or more performant from using intrinsics or asm.

This makes sense when you think about it. The compiler would either need to be extremely capable of generalizing about some kinds of arithmetic, or it would need millions of special cases to recognize. By writing your own vectorization, you are basically covering your singular special case on your own.

Re: Towards fearless SIMD

#49

Post by Const-me » Wed, Oct 24, 2018, 8:36 PM UTC

marmaduke wrote: post #43

Const-me wrote: post #30

Earlier quoted context omitted.

> it is quite useful if the compiler can SROA __attribute__((always_inline)) / __forceinline usually help. > LICM (you're calling a function that needs a constant vector in a loop) I can calculate that vector outside of the loop. > link in an assembly file. Way more complex, I need to write outer scalar code in assembly too, need to manually allocate registers, also C++ templates are sometimes very useful for that ki…

Can’t just isolate those in a compilation unit with -O0?

Interesting idea, will try next time.

I usually want to optimize the scalar code outside of the manually-vectorized body of the loops. A function call to that external compilation unit will be slower than inlining I have when everything is in the same unit. However, it could be the call overhead is small enough, obviously need to profile.

Re: Towards fearless SIMD

#50

Post by pjmlp » Wed, Oct 24, 2018, 8:47 PM UTC

gameswithgo wrote: post #33

pjmlp wrote: post #13

Earlier quoted context omitted.

Intel has made quite a few contributions to Hotspot, including AVX support. ART started supporting SIMD on Oreo, and it was further expanded on Pie. Naturally very few devices have gotten those improvements thanks the state of Android's updates. So on Android's case Renderscript is still the best way for a JIT like approach for SIMD.

As far as I know the JVM will only auto-vectorize integers. Is there a way to tag a function such that you want floats vectorized too now? ART is a good point, compile on install lets you do this.

You can get the information here. It also refers to floating point calculations.

http://cr.openjdk.java.net/~vlivanov/talks/2017_Vectorizatio...

https://software.intel.com/sites/default/files/managed/19/ae...

Also the Vector API development is ongoing and there is a talk at this week's Oracle ONE about the current state.

ART no longer compiles on install since Android 7, that behaviour is specific to Android 5 and 6 versions.

Since 7 it is a multistage runtime with hand written in Assembly interpreter, JIT + PGO, AOT + PGO on idle device. And as of 9, PGO data gets uploaded into the store and shared across devices on installation.