The Curious Case of MD5 • Hacker News Board

Re: The Curious Case of MD5

#171

Post by marshray » Tue, Jan 09, 2024, 5:53 AM UTC

StillBored wrote: post #169

marshray wrote: post #76

Earlier quoted context omitted.

> Md5 is just as fine as any other generic hash function if its being used as a non-unique key MD5 brings the feature that you'll forever be explaining why you chose a function that had already been broken for 30 years when other options were readily available.

Often, its less about picking it for a new project vs having the discussion about how to update an existing one, often with data at rest that needs converting. In the latter case often the hash needs to fit into the existing 128b field size, so one is throwing a good number of the SHA bits away anyway.

This is not a justification for using a broken function.

Truncating a fixed number of bits does not make a good secure function less secure, other than the implications that the shorter length has on brute-force and collision strength.

In some cases it can even make it even more secure, e.g. SHA-2-512/256.

Re: The Curious Case of MD5

#172

Post by gojomo » Fri, Jan 12, 2024, 1:41 AM UTC

AlienRobot wrote: post #168

gojomo wrote: post #165

Earlier quoted context omitted.

Conventional analysis of non-contrived files would suggest only a 50% chance of an MD5 collision among 2^64 (18 quintillion) files. So any SO account of "hash collisions in MD5 (128 bit) after hashing around 20k files" may be indicative of some other bug, misreporting, or having a set of files that includes pairs intentionally-contrived to include MD5 collisions. (That's now trivial as long as you're not targeting a…

I'm not well-versed in this, but I think you're wrong. A MD5 hash has 2^128 permutations, so it's like the birthday problem, you'll get 50% chance of any two files in a set having the same hash with far fewer than 2^64 files. Besides, we're talking about probabilities. It's possible for 2 files, one after the other, to have the exact same hash just because of chance. In fact, I think there was a bitcoin bug due to th…

Nope. 2^64 is the number of tries needed for a 50% chance of a collision between two of the tries. You don't have to stay poorly read on this, and groundlessly allege wrongness:

https://auth0.com/blog/amp/birthday-attacks-collisions-and-p...

Re: The Curious Case of MD5

#173

Post by StillBored » Mon, Jan 22, 2024, 8:59 PM UTC

marshray wrote: post #171

StillBored wrote: post #169

Earlier quoted context omitted.

Often, its less about picking it for a new project vs having the discussion about how to update an existing one, often with data at rest that needs converting. In the latter case often the hash needs to fit into the existing 128b field size, so one is throwing a good number of the SHA bits away anyway.

This is not a justification for using a broken function. Truncating a fixed number of bits does not make a good secure function less secure, other than the implications that the shorter length has on brute-force and collision strength. In some cases it can even make it even more secure, e.g. SHA-2-512/256.

Uh, truncation is a huge no-no for security. But, that isn't what I was talking about. For deduplication/hash key/etc kinds of functions where a checksum works (or some other algorithm with a high chance of collisions is implemented for unit testing to assure collision paths are handled correctly).

This is part of the entire discussion, it's possible to use algorithms originally intended for cryptographically secure purposes in places where the reversibility/collision properties aren't considered problems. And likely this will only happen more frequently as algorithms are picked because they have hardware acceleration and are fast rather for the underlying security properties.

BTW: There is a fun game that I've played; see how much you can truncate a modern cryptographic hash before it becomes trivial to find collisions.