Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Accelerating SHA256 by 100x in Golang on ARM

blog.minio.io

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#61

Post by pbsd »

kalden_ wrote: post #30
mrb wrote: post #24

"Interestingly enough, there are actually comparable Intel SHA extensions to the ARM equivalents. Linux 4.4 has added support for this but so far we have not been able to identify any CPUs that will actually run this code." Intel Goldmont is the only microarch that implements the instructions. It was "released" in April: http://www.extremetech.com/computing/226800-intels-new-low-c... But it is one of these annoying s…

It seemed to me that some Xeons had those since 2014. See https://software.intel.com/en-us/isa-extensions/intel-sha Should something be understood instead?

I think even Intel is confused about their own extensions at this point. The E7 and E5 Xeons are Haswell systems that slightly speed up SHA-256 and 512 with the introduced BMI2 RORX instruction.

As far as I know there are no SHA extensions before Goldmont, as mrb said, and in Cannonlake for non-low-power chips.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#62

Post by DanielDent »

akovaski wrote: post #51
DanielDent wrote: post #40

Earlier quoted context omitted.

That fits with my understanding. What I don't know is how this interacts with the acceleration functionality the chip has. For instance, do the instructions end up being "please do sha256 on this data", or are they closer to AES-NI where it's "please perform one round of an AES encryption flow".

Judging from the source code[1] in the author's repository, it is the latter. (what I see is: perform 96 instructions, then subtract 64 from the message length, and repeat until the message length is 0) Intel seems to be the same way [2](2013) - i.e. looping over multiple instructions per 64 byte block. [1] https://github.com/minio/sha256-simd/blob/master/sha256block... [2] https://software.intel.com/en-us/articles/i…

Interesting, thanks for pointing that out.

This code is such a good example of the potential value of formal verification tools for crypto primitives.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#63

Post by sjolsen »

halomru wrote: post #56
fwessels wrote: post #16

Earlier quoted context omitted.

No, there are no algorithm changes (otherwise the results would be different), it is just taking advantage of the ARM SHA accelerations when they are available.

>there are no algorithm changes (otherwise the results would be different Insertionsort, Mergesort and Timsort are three wildly different algorithms with different speeds, but on every possible input they produce the exact same result

Strictly speaking, it's possible for different sorting algorithms to produce different results if you sort using a weak order rather than a total order.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#64

Post by pbsd »

tedunangst wrote: post #59
cyphar wrote: post #37

Earlier quoted context omitted.

The oddity is that I believe sha512 uses 64-bit words while sha256 uses 32-bit words. So on 64 bit hardware sha512 will be faster (presumably because using the 32 bit registers is slower somehow, I don't know). This is why ZFS is adding support for sha512/256 as it's Merkel tree hashing function.

SHA512 eats data 512 bits at a time, while SHA256 eats it 256 bits at time. Both internally use 8 "registers", which are either 64 or 32 bits wide. Assuming you have the hardware registers to match, this would make SHA512 about twice as fast. But internally, it mixes data using 80 rounds, vs 64 for SHA256, so the speedup isn't quite 2x.

For the sake of pedantry, SHA-512 eats 1024 bits, resp. 512 bits for SHA-256, at a time. It's the chaining variables that are of those lengths.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#65

Post by pcwalton »

giovannibajo1 wrote: post #31
white-flame wrote: post #29

Earlier quoted context omitted.

From a bird's eye view, adjust the compiler until it outputs the assembly you would have written. Hopefully this is done in such a way that such adjustments can be cross-beneficial across platforms, and can be easily created.

That's wishful thinking. GCC isn't able to do that with C language, a 20 years old compiler with a 40 years language. How would a compiler know that you can use a SHA2 instruction? Yes, there are intrinsics but then what about specialized vectorized instructions which don't match any primitive types or operations in the language (like, you can't express "add with carry" in C or Go)? And how can you explain a compiler…

> How would a compiler know that you can use a SHA2 instruction?

Intrinsics.

There's basically zero benefit to coding this stuff in assembly. Many standard libraries prefer intrinsics, because there's basically zero benefit to writing this stuff in assembly and many drawbacks from an optimization perspective.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#66

Post by pcwalton »

giovannibajo1 wrote: post #22
theparanoid wrote: post #20

The amount of assembly in the Go ecosystem is crazy, ~1% of the standard library is assembly. There are better ways.

Most of it is written to take advantage of special hardware instructions in CPUs. If you take OpenSSL (a C library), it's got many assembly code paths as well. I'm not sure why this should be a problem, it just shows attention to performance in my opinion.

Opaque assembly blocks that the compiler doesn't understand the effects of are actually suboptimal for performance.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#67

Post by userbinator »

This is the sort of thing that shows why the whole "pure RISC philosophy" of implementing only the simplest instructions is ultimately a pretty dead-end in processor design. CISC-style dedicated instructions and hardware which utilises them will always be more efficient than a pure software implementation using just "simple RISC instructions", because hardware can easily express specialised computations like the extreme parallelism of algorithms such as SHA while software implementations essentially rely on the instruction-scheduling mechanisms to arrange potentially hundreds or even thousands of instructions in an optimal order.

In addition, when there is dedicated hardware, it is also significantly easier to add a SHA instruction than attempt to recognise sequences of many regular instructions that could be "fused together" into a single operation using that hardware. When there isn't, adding such an instruction that gets internally expanded into multiple uops for the equivalent using the existing functional units is still beneficial, since it leaves open room for an immediate performance gain on all existing software when a future revision does add that dedicated hardware, or optimises its microarchitecture to allow those (possibly different) uops to execute faster.

Despite the name, ARM is definitely not very RISC anymore, and that's what has kept it competitive.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#68

Post by tedunangst »

cperciva wrote: post #60
tedunangst wrote: post #58

Earlier quoted context omitted.

I waffle about with this metric. If it's cheap to buy a server with some feature, it's obviously cheap for an attacker to buy the same. Widespread SHA2 hardware means that SHA2 cracking hardware is also available to even the poorest of crackers. I think there's a sliding scale, where the ratio changes for some attackers but not for all.

Right, sophisticated attackers have had custom SHA256 circuits for a long time; less sophisticated attackers are only gaining them now that they are present in CPUs. But if you're defending against such less sophisticated attackers, you're still not losing anything; worst case, SHA256 instructions in CPUs give them the same speedup as you're getting so their cost remains fixed.

The question is who upgrades their hardware faster, attackers or big enterprises? :)

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#69

Post by Buge »

DanielDent wrote: post #34

Did you compare performance to SHA512? Despite being a theoretically more secure/"harder" algorithm, on 64 bit platforms it can sometimes be faster than SHA256. If you don't want to use 512 bits, using 256 bits of the output of SHA512 is standardized as SHA512/256 and is considered valid/secure. (I'm unclear if this performance oddity remains true with the crypto hardware extensions being used here.)

SHA512/256 is also secure against length extension, which SHA256 isn't.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#70

Post by drchickensalad »

pcwalton wrote: post #66
giovannibajo1 wrote: post #22

Earlier quoted context omitted.

Most of it is written to take advantage of special hardware instructions in CPUs. If you take OpenSSL (a C library), it's got many assembly code paths as well. I'm not sure why this should be a problem, it just shows attention to performance in my opinion.

Opaque assembly blocks that the compiler doesn't understand the effects of are actually suboptimal for performance.

Generally yes, specialized no.
Post reply on HN