Live data from Hacker News

Poisonous MD5 – Wolves Among the Sheep

blog.silentsignal.eu

21–30 of 55 posts

Re: Poisonous MD5 – Wolves Among the Sheep

#21
post #7
post #5

Earlier quoted context omitted.

Your parenthetical is the key, though. There's a big difference between a hash algorithm where generating a collision requires a few minutes of work on a cheap computer (MD5, now) and a hash algorithm where generating a collision requires a computer the size of the universe operating for a trillion trillion years (any good cryptographically secure hash).

Cool - didn't realize the difference was so great. I've always known that the good algorithms are better because they're more difficult to brute-force, but always wondered if it's just a matter of a few years before the "impossible" becomes possible. Your illustration helps clarify that improbability in my mind - thanks!

Glad to be of service. This seems to be a common misconception.

To put some numbers on it, a good cryptographic hash should produce random-looking output with no way to predictably influence the output, and no visible correlation between the input and the output. Put in A, get out "random number" B, except that every time you put in the same A, you get out the same B.

If you have a hash that matches that description, then your only hope is brute force, so you look at the size of the hash. If you're breaking SHA-256, for example, that's 256 "random" bits.

If you have a specific hash that you want to generate, it will take you on average 2^255 attempts. (On average you have to try half the possibilities before you find a match.) If your computer can do a billion hashes per nanosecond then that means you'll spend on average 2^255 / (10^9 * 10^9) seconds, or about 10^41 times the current age of the universe.

If you just want to find any collision, then the birthday paradox comes into play, and on average you'll need to try roughly the square root of the number of possibilities before you find a collision. At a billion per nanosecond that's 2^128 / (10^9 * 10^9) second, or a mere 780 times the current age of the universe. Plus you have to figure out how to store all those intermediate results.

If MD5 fit this property it would still be good. Not as good as SHA-256, because it's only 128 bits, but plenty sufficient. 128 bits with the hypothetical billion hashes per nanosecond gives you 390 times the current age of the universe to find a specific hash. A collision is easier, at 18 seconds, but a billion hashes per nanosecond is also orders of magnitude faster than you'll realistically be able to do, and you'll need 2^64 * 128 bits = 300 exabytes of storage.

The problem with MD5 is that it does not fit this "random number" property. You can manipulate the input with somewhat predictable results in the output if you're clever, and that means you can generate a collision much easier than it would require for brute force.

My understanding is that more modern hashes are believed/hoped to have this property, but it's unknown. And not only that, but it's unknown whether any hash could exist with that property, or whether it's a theoretically impossible goal.

Re: Poisonous MD5 – Wolves Among the Sheep

#22
post #4
post #2

Aren't all hashing algorithms vulnerable to the possibility for collisions (albeit with different degrees of difficulty)? It sounds like the problem here is more related to the logic that relies on a hash alone to make important decisions. Not saying that MD5 is a good choice in this case, just that we may be blaming the wrong thing.

The collision-resistance property that all good hashes should have (and md5 lacks) states that an attacker with an input and its hash cannot arbitrarily produce a second input with the same hash. The possibility of it happening in the wild will always exist with hashes by their finite nature, but the only way an attacker should be able to find collisions is by enumerating the input space (rainbow table generation).

No, the property you describe is called "preimage resistance". Collision resistance is stronger; it states that an attacker should not be able to create a pair of inputs with the same hash. In the case of md5, creating a pair of inputs with the same hash is easier than creating another input with the same hash as something else which you didn't yourself generate.

The MD5 algorithm is known to lack collision resistance, but whether it has preimage resistance is less certain; mathematical advances have weakened its preimage resistance, but not yet to the point of demonstrating a practical preimage attack.

Re: Poisonous MD5 – Wolves Among the Sheep

#23
post #10
post #5

Earlier quoted context omitted.

Your parenthetical is the key, though. There's a big difference between a hash algorithm where generating a collision requires a few minutes of work on a cheap computer (MD5, now) and a hash algorithm where generating a collision requires a computer the size of the universe operating for a trillion trillion years (any good cryptographically secure hash).

Did breaking MD5 require a computer the size of the universe 20 years ago?

Not really. It was cracked by a distributed network of ordinary computers were people volunteered CPU time.

Re: Poisonous MD5 – Wolves Among the Sheep

#24
post #6

I don't get why it is a security problem that someone can manufacture false positives for an anti-virus. What is the benefit for a virus to have non-malicious code caught by the anti-virus? False negatives would be more of an issue if the anti-virus has white lists and one can manufacture a Microsoft Excel MD5 signature with a malware. But that's not what the article refers to. MD5 is only broken if you want to use i…

You misunderstand. The researchers are presenting a way to manufacture false negatives for an anti-virus. It works by confusing antivirus vendors' infrastructure into thinking it's already analyzed an executable and found it to be innocent when it's really analyzed something else.

Re: Poisonous MD5 – Wolves Among the Sheep

#25

Earlier quoted context omitted.

Hmmm... so the takeaway for me is that when designing critical systems that rely on hashes/fingerprints as identifiers, we should probably treat those as transient identifiers with a reasonable expectation of migrating them as newer algorithms replace older broken ones. Does that sound right?

That's correct.

I feel like this gets missed in the debate over the "best" hash algorithms. It seems like the message is always "use XYZ algorithm for everything - it's practically perfect and the only one you'll ever need". In reality, it should be more along the lines of "migrate to this one for now, but don't build things that depend on it being the best option next year since it's probably not perfect. Design your systems to easily accommodate changes to the algorithm."

Re: Poisonous MD5 – Wolves Among the Sheep

#26
post #21
post #7

Earlier quoted context omitted.

Cool - didn't realize the difference was so great. I've always known that the good algorithms are better because they're more difficult to brute-force, but always wondered if it's just a matter of a few years before the "impossible" becomes possible. Your illustration helps clarify that improbability in my mind - thanks!

Glad to be of service. This seems to be a common misconception. To put some numbers on it, a good cryptographic hash should produce random-looking output with no way to predictably influence the output, and no visible correlation between the input and the output. Put in A, get out "random number" B, except that every time you put in the same A, you get out the same B. If you have a hash that matches that description,…

Also, even it were provably true that some hash algorithm has this property, we would have to implement it properly. That requires the algorithm itself as well as all of its dependencies to be implemented properly. It's very easy to make an implementation mistake that breaks security.

Re: Poisonous MD5 – Wolves Among the Sheep

#27
post #4

Earlier quoted context omitted.

The collision-resistance property that all good hashes should have (and md5 lacks) states that an attacker with an input and its hash cannot arbitrarily produce a second input with the same hash. The possibility of it happening in the wild will always exist with hashes by their finite nature, but the only way an attacker should be able to find collisions is by enumerating the input space (rainbow table generation).

No, the property you describe is called "preimage resistance". Collision resistance is stronger; it states that an attacker should not be able to create a pair of inputs with the same hash. In the case of md5, creating a pair of inputs with the same hash is easier than creating another input with the same hash as something else which you didn't yourself generate. The MD5 algorithm is known to lack collision resistanc…

> In the case of md5, creating a pair of inputs with the same hash is easier than creating another input with the same hash as something else which you didn't yourself generate.

This is the case with all instances of seeking a collision, due to the birthday paradox [0]

0: https://en.wikipedia.org/wiki/Birthday_attack

Re: Poisonous MD5 – Wolves Among the Sheep

#28

Earlier quoted context omitted.

That's correct.

I feel like this gets missed in the debate over the "best" hash algorithms. It seems like the message is always "use XYZ algorithm for everything - it's practically perfect and the only one you'll ever need". In reality, it should be more along the lines of "migrate to this one for now, but don't build things that depend on it being the best option next year since it's probably not perfect. Design your systems to eas…

Correct. http://valerieaurora.org/hash.html

Re: Poisonous MD5 – Wolves Among the Sheep

#29
post #4

Earlier quoted context omitted.

The collision-resistance property that all good hashes should have (and md5 lacks) states that an attacker with an input and its hash cannot arbitrarily produce a second input with the same hash. The possibility of it happening in the wild will always exist with hashes by their finite nature, but the only way an attacker should be able to find collisions is by enumerating the input space (rainbow table generation).

No, the property you describe is called "preimage resistance". Collision resistance is stronger; it states that an attacker should not be able to create a pair of inputs with the same hash. In the case of md5, creating a pair of inputs with the same hash is easier than creating another input with the same hash as something else which you didn't yourself generate. The MD5 algorithm is known to lack collision resistanc…

My mistake, I always mixed those two up. Both properties address OP, though, as MD5 is not suspected to have preimage resistance either (it's just not to the point of somebody having done it yet).

Re: Poisonous MD5 – Wolves Among the Sheep

#30

Earlier quoted context omitted.

That's correct.

I feel like this gets missed in the debate over the "best" hash algorithms. It seems like the message is always "use XYZ algorithm for everything - it's practically perfect and the only one you'll ever need". In reality, it should be more along the lines of "migrate to this one for now, but don't build things that depend on it being the best option next year since it's probably not perfect. Design your systems to eas…

That is the common wisdom. However some people, most known perhaps djb, actually argue against it. Pluggable algorithms may allow for downgrade attacks, and in a world wide complex system there will always be holdouts for some reason.

They argue you would be better with hardcoding your system to known secure best practice. When the time comes to change it, you specify a new protocol version, as there will be new and better practices not only in algorithms but also in how they are used (mac-before-encryption being the canonical example of this which took far too long to change).

Post reply on HN