Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The long tail of MD5

tedunangst.com

Post reply on HN
Print view View original

31–40 of 58 posts

Re: The long tail of MD5

#31

Post by ketralnis »

smackfu wrote: post #27
jonstewart wrote: post #16

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashes. * This is not an endorsement of MD5 for such use cases.

Do the security problems with MD5 make it bad as a hashing algorithm?

Not unless hash collisions are also a security issue (see http://bugs.python.org/issue13703). But it's much slower than a traditional hashtable hashing function and not much slower than a sha family hash.

Maybe that's the sweet spot for your project, but I think most non-security uses of MD5 are because it already exists in most languages and is pretty universal

Re: The long tail of MD5

#32

Post by v64 »

tptacek wrote: post #23
v64 wrote: post #21

Earlier quoted context omitted.

I believe the spirit of the original poster's message was that SipHash is not strong enough to be used for cryptographic purposes.

No, SipHash is acceptable cryptographic PRF; it's just not acceptable as a standalone general-purpose cryptographic hash. A PRF is secure as long as attackers don't get to know exactly how it works, which, with SipHash, they don't if they don't know the key used for it. A cryptographic hash function is secure --- it's Hard to find two inputs that produce the same output, it's (even) Hard(er) to find a specific input…

Makes sense, thanks for the clarification.

Re: The long tail of MD5

#33

Post by MagicWishMonkey »

jonstewart wrote: post #16

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashes. * This is not an endorsement of MD5 for such use cases.

Are collisions really that much of a concern (other than for security reasons)? IIRC they are still incredibly rare even with older hashing algorithms like MD5.

Re: The long tail of MD5

#34

Post by MagicWishMonkey »

WhitneyLand wrote: post #25
Andys wrote: post #17

Earlier quoted context omitted.

Slow? I thought md5 was faster than the SHA-* family

Yes, way faster. It would be great if it weren't for those pesky collisions.

Collisions aren't a concern unless you're hashing trillions of records. It's not really a concern for the vast majority of software projects.

Re: The long tail of MD5

#35

Post by nly »

MagicWishMonkey wrote: post #33
jonstewart wrote: post #16

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashes. * This is not an endorsement of MD5 for such use cases.

Are collisions really that much of a concern (other than for security reasons)? IIRC they are still incredibly rare even with older hashing algorithms like MD5.

Collisions render hash functions useless as a component of most (but not all) digital signature algorithms

Re: The long tail of MD5

#36

Post by wbl »

lisper wrote: post #22
pbsd wrote: post #19

Earlier quoted context omitted.

No, I meant what I wrote. SipHash is fine as a keyed primitive -- 128-bit security against key recovery, (64 - t)-bit security against 2^t forgery attempts. However, as a hash function (e.g., with a fixed key), it is entirely too weak: 2^32 work for a collision, and 2^64 work for arbitrary (second-)preimages.

OK... so if it's suitable as a PRF, then you should be able to widen the output by using two different keys and concatenating the results, no?

Unfortunately not: it's only 64 times slower to generate 2^64 collisions for hash function by generic attacks than just one, and those 2^64 messages will contain a collision for another hash function of length 128. The total time is only the sum of the times for each one.

Re: The long tail of MD5

#37

Post by vertex-four »

MagicWishMonkey wrote: post #34
WhitneyLand wrote: post #25

Earlier quoted context omitted.

Yes, way faster. It would be great if it weren't for those pesky collisions.

Collisions aren't a concern unless you're hashing trillions of records. It's not really a concern for the vast majority of software projects.

Collisions are a concern if an attacker has any input into what you're hashing, and can therefore induce collisions. MD5 is broken in that there are real-world attacks by which an attacker can do so.

Re: The long tail of MD5

#38

Post by IvyMike »

Want to kill MD5?

Provide public-domain easy-to-compile/use versions for all languages, and furthermore, get their google page ranks high.

Do not underestimate laziness. If Joe Random can find a suitable MD5 algorithm in 10 seconds but it takes 30 seconds to find a suitable SHA algorithm, guess which one gets used?

Re: The long tail of MD5

#39

Post by lisper »

wbl wrote: post #36
lisper wrote: post #22

Earlier quoted context omitted.

OK... so if it's suitable as a PRF, then you should be able to widen the output by using two different keys and concatenating the results, no?

Unfortunately not: it's only 64 times slower to generate 2^64 collisions for hash function by generic attacks than just one, and those 2^64 messages will contain a collision for another hash function of length 128. The total time is only the sum of the times for each one.

Ah, that makes sense. Thanks!

Re: The long tail of MD5

#40

Post by dredmorbius »

smackfu wrote: post #27
jonstewart wrote: post #16

MD5 also remains quite popular for file hashing/hash set analysis. I.e., is this file a member of a known set? For example, NIST's NSRL still includes MD5 hashes. * This is not an endorsement of MD5 for such use cases.

Do the security problems with MD5 make it bad as a hashing algorithm?

Do you know the contexts your hashing requirement is likely to be used in, and/or may be adapted to?

The worst and most persistent security problems emerge when someone defends half-assery as acceptable because, take your pick, it's a quick hack / it's a personal project / it's a small project / it's not used for critical infrastructure.

Until eventually is one or more of the above are violated.

The primary advantage of MD5 is that the hashes are (slightly) shorter than those of other checksum methods. This makes it slightly more convenient to manually compare or transmit hashes.

My problem is that I happen to have used md5sum so often and for so long that it's wired into my own wetware and muscle-memory, and I'm not sure which of the alternative SHA sums I should use, and which of those are widely available. I honestly don't know the answer to that off the top of my head, and "what to use instead of md5sum" as a DDG or Google search doesn't turn up a clearly useful answer. "sha1 sha256 sha384 sha512 which to use" does better.

And locally I've got utilities for SHA 1, 224, 256, and 512 installed, that I can tell.

Looks as if SHA-2 and SHA-3, which include keylengths of 224-512, are considered secure:

http://en.wikipedia.org/wiki/Secure_Hash_Algorithm

Then there are some openssl utilities. But let's not go there.

Post reply on HN