Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Accelerating SHA256 by 100x in Golang on ARM

blog.minio.io

Post reply on HN
Print view View original

51–60 of 93 posts

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#51

Post by akovaski »

DanielDent wrote: post #40
cyphar wrote: post #37

Earlier quoted context omitted.

The oddity is that I believe sha512 uses 64-bit words while sha256 uses 32-bit words. So on 64 bit hardware sha512 will be faster (presumably because using the 32 bit registers is slower somehow, I don't know). This is why ZFS is adding support for sha512/256 as it's Merkel tree hashing function.

That fits with my understanding. What I don't know is how this interacts with the acceleration functionality the chip has. For instance, do the instructions end up being "please do sha256 on this data", or are they closer to AES-NI where it's "please perform one round of an AES encryption flow".

Judging from the source code[1] in the author's repository, it is the latter. (what I see is: perform 96 instructions, then subtract 64 from the message length, and repeat until the message length is 0)

Intel seems to be the same way [2](2013) - i.e. looping over multiple instructions per 64 byte block.

[1] https://github.com/minio/sha256-simd/blob/master/sha256block... [2] https://software.intel.com/en-us/articles/intel-sha-extensio...

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#52

Post by pbsd »

y4m4b4 wrote: post #41
DanielDent wrote: post #34

Did you compare performance to SHA512? Despite being a theoretically more secure/"harder" algorithm, on 64 bit platforms it can sometimes be faster than SHA256. If you don't want to use 512 bits, using 256 bits of the output of SHA512 is standardized as SHA512/256 and is considered valid/secure. (I'm unclear if this performance oddity remains true with the crypto hardware extensions being used here.)

For comparison with other checksums you should look here https://github.com/minio/sha256-simd#comparison-to-other-has...

SHA-512 should be much faster than SHA-256 on AVX2-capable processors. Go's implementation is subpar; try OpenSSL instead. On Skylake, SHA-512 is about as fast as MD5 now -- https://bench.cr.yp.to/results-hash.html#amd64-skylake

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#53

Post by y4m4b4 »

smegel wrote: post #44

> minio/sha256-simd. So rather than writing your own module and a blog post bragging about it, why not just send a pull request to Golang core? There are lots of assembly code in there already dealing with special instructions on different architectures, there is nothing special about this. Or is being a humble programmer the antithesis of startup culture?

This is the plan, which will happen eventually right now we are keeping it out there for community comments and improvement in quality that might be needed before it is submitted upstream.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#54

Post by y4m4b4 »

mrb wrote: post #24

"Interestingly enough, there are actually comparable Intel SHA extensions to the ARM equivalents. Linux 4.4 has added support for this but so far we have not been able to identify any CPUs that will actually run this code." Intel Goldmont is the only microarch that implements the instructions. It was "released" in April: http://www.extremetech.com/computing/226800-intels-new-low-c... But it is one of these annoying s…

Thanks this is a nice share, we are definitely looking forward to this.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#56

Post by halomru »

fwessels wrote: post #16
wyldfire wrote: post #3

... by moving from a software implementation to a mostly hardware one. ;) Still a worthwhile article but the title seemed to make me think of implementation/algorithm changes.

No, there are no algorithm changes (otherwise the results would be different), it is just taking advantage of the ARM SHA accelerations when they are available.

>there are no algorithm changes (otherwise the results would be different

Insertionsort, Mergesort and Timsort are three wildly different algorithms with different speeds, but on every possible input they produce the exact same result

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#57

Post by y4m4b4 »

pbsd wrote: post #52
y4m4b4 wrote: post #41

Earlier quoted context omitted.

For comparison with other checksums you should look here https://github.com/minio/sha256-simd#comparison-to-other-has...

SHA-512 should be much faster than SHA-256 on AVX2-capable processors. Go's implementation is subpar; try OpenSSL instead. On Skylake, SHA-512 is about as fast as MD5 now -- https://bench.cr.yp.to/results-hash.html#amd64-skylake

Yes true. OpenSSL version gives a great boost. Rather than SHA512 we ended up using Blake2b - https://github.com/minio/blake2b-simd (also optimized with SIMD instructions) internally for bit-rot verification in https://github.com/minio/minio

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#58

Post by tedunangst »

cperciva wrote: post #43
niftich wrote: post #21

Earlier quoted context omitted.

It's fine to use a SHA2 variant HMAC in PBKDF2 with an obscene, ever-increasing number of rounds, but the scrypt paper [1] provides a detailed justification why you may want to switch to a dedicated password hashing function regardless. To quote their website [2]: "We estimate that on [circa-2009] hardware, if 5 seconds are spent computing a derived key, the cost of a hardware brute-force attack against scrypt is rou…

Almost right. bcrypt is not designed to be secure against ASICs; it requires a fixed circuit size. In fact, given that CPU acceleration is available for SHA2, I suspect that PBKDF2-SHA256 is now stronger than bcrypt based on the "if my server spends X seconds hashing this password, how much money will someone need to spend to crack it" metric.

I waffle about with this metric. If it's cheap to buy a server with some feature, it's obviously cheap for an attacker to buy the same. Widespread SHA2 hardware means that SHA2 cracking hardware is also available to even the poorest of crackers. I think there's a sliding scale, where the ratio changes for some attackers but not for all.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#59

Post by tedunangst »

cyphar wrote: post #37
DanielDent wrote: post #34

Did you compare performance to SHA512? Despite being a theoretically more secure/"harder" algorithm, on 64 bit platforms it can sometimes be faster than SHA256. If you don't want to use 512 bits, using 256 bits of the output of SHA512 is standardized as SHA512/256 and is considered valid/secure. (I'm unclear if this performance oddity remains true with the crypto hardware extensions being used here.)

The oddity is that I believe sha512 uses 64-bit words while sha256 uses 32-bit words. So on 64 bit hardware sha512 will be faster (presumably because using the 32 bit registers is slower somehow, I don't know). This is why ZFS is adding support for sha512/256 as it's Merkel tree hashing function.

SHA512 eats data 512 bits at a time, while SHA256 eats it 256 bits at time. Both internally use 8 "registers", which are either 64 or 32 bits wide. Assuming you have the hardware registers to match, this would make SHA512 about twice as fast. But internally, it mixes data using 80 rounds, vs 64 for SHA256, so the speedup isn't quite 2x.

Re: Show HN: Accelerating SHA256 by 100x in Golang on ARM

#60

Post by cperciva »

tedunangst wrote: post #58
cperciva wrote: post #43

Earlier quoted context omitted.

Almost right. bcrypt is not designed to be secure against ASICs; it requires a fixed circuit size. In fact, given that CPU acceleration is available for SHA2, I suspect that PBKDF2-SHA256 is now stronger than bcrypt based on the "if my server spends X seconds hashing this password, how much money will someone need to spend to crack it" metric.

I waffle about with this metric. If it's cheap to buy a server with some feature, it's obviously cheap for an attacker to buy the same. Widespread SHA2 hardware means that SHA2 cracking hardware is also available to even the poorest of crackers. I think there's a sliding scale, where the ratio changes for some attackers but not for all.

Right, sophisticated attackers have had custom SHA256 circuits for a long time; less sophisticated attackers are only gaining them now that they are present in CPUs. But if you're defending against such less sophisticated attackers, you're still not losing anything; worst case, SHA256 instructions in CPUs give them the same speedup as you're getting so their cost remains fixed.
Post reply on HN