Your code is fast if you're lucky

A large part of optimization is understanding the hardware architecture in detail and then making your software architecture mirror that hardware architecture as closely as possible. A compiler can't do this for you. Much of "performance engineering" is applying your understanding of how the hardware components work and are connected to the software design.

Most software introduces a large number of unnecessary stalls where one part of the hardware is waiting on or bottlenecked by a different piece of hardware. Optimization is often about removing or minimizing these stalls to the extent possible. But to do this you need to understand why specific code choices cause the hardware to stall in the first place. The most basic form of this is CPU cache locality but there are many levels.

Compilers are great at localized micro-optimizations, except for SIMD. Most idiomatic code can be detected and transformed by the compiler into something that takes advantage of the hardware design. You don't need to do tricky bit-twiddling stuff, the compiler is better at it than you are in most cases (this wasn't always true). I always recommend people interested in optimization experiment running snippets of code through the compiler using godbolt with optimization turned on. You can learn a lot about how the compiler sees and understands your code by studying this. It is also a good way to became familiar with assembly.

Leveraging SIMD and vector ISAs is a mess. Different hardware architectures rely on different idioms and compilers cannot auto-vectorize most code that can be vectorized. You need to learn the idioms of each vector ISA and how to write them using intrinsics. A great way to learn this is reading other peoples' SIMD code. Modern SIMD code is wide in that you need to memorize a lot of things but conceptually pretty shallow. It mostly comes down to learning the idiomatic tricks and gadgets for an ISA -- code is composed from these conceptual primitives.

Totally agree, but I'm in Java land and cursed to have about the worst case scenario for those things :D

Oh yeah I'm also mostly in Java land. These concerns are also concerns in Java, plus whatever C2 gets up to.

Valhalla can't come soon enough.

I've definitely had to change things into SOA in order to eke out performance. My coworkers aren't thrilled seeing `double[] x; double[] y;` but that really is about the only way to get the JVM to play nice.

Representing code in a compiler is not precisely trivial, and the two statements are actually quite different from a compiler or AST perspective. Just looking at the first branch:

  *lwr = x; lwr++;

This could be be represented with something like this (and this is a very vague approximation of an AST):

  block
    statement (assignment)
      expression
        operator (dereference)
          variable
      expression
        variable
    statement
      expression
        operator (post-increment)
          variable

The second form, that looks this in the source:

  *lwr++ = x

might look like this in the AST:

  block
    statement (assignment)
      expression
        operator (post-increment)
          operator (dereference)
            variable
      expression
        variable
 

If these two ASTs are to take the same form in the compilation pipeline, there needs to be some kind of pass that transforms one into the other.

As for why the second form is faster (or rather, why the compiler can generate faster code): There is likely an optimisation pass somewhere in llvm that recognises a pattern that this fits into, which allows it to generate branchless instructions. For instance, in the second form, there is a pattern of:

  operator (post-increment)
    operator (dereference)

that might be recognised by a pass. In the former form, the two operators are far apart in the tree, so a pass would have to "look further" to match them up. A single pass likely won't do this, either for (compiler) performance reasons, or for correctness reasons.

Finding which pass that is can be non-trivial, as it's more than a matter of enabling individual passes until one works. It might be that an earlier pass does some code reshaping that allows the relevant pass to work. My suggestion would be to dump the llvm ir at the end, and find the rough pattern that you're looking for, then re-run the compilation with `-mllvm -print-after-all` to see what the IR looks like after each pass, and then manually "look back" until you can't see the pattern any more.