Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Curious Case of MD5

katelynsills.com

Post reply on HN
Print view View original

61–70 of 173 posts

Re: The Curious Case of MD5

#61

Post by bawolff »

a1369209993 wrote: post #30
Dwedit wrote: post #29

Earlier quoted context omitted.

Don't be silly here, the MD5 is clearly in the plaintext here, and the NES ROM is only the first 40k of the file. It is not able to scan itself and print out a hash that way.

> the MD5 is clearly in the plaintext here Alright, I'll bite: at what byte offset in the binary file contents does a trivial encoding[0] of the MD5 hash occur? > the NES ROM is only the first 40k of the file. It is not able to scan itself and print out a hash that way. It is possible to encode the effects of multiple blocks of arbitrary[1] data on a hash function internal state (independently of what state you start…

I dont know about that specific file, but pdf usually gzips its component parts so i wouldn't expect naive grepping to work.

Re: The Curious Case of MD5

#62

Post by tgamblin »

The article mentions the key detail: MD5 is broken for cryptography (collisions) but not for second preimage attacks. I was hoping there would be some discussion of just how much more difficult the latter is. It is extremely difficult.

Let’s ignore that no second preimage attack is currently known for MD5. The software the author links to has a FAQ that links to a paper that lays out the second preimage complexity for MD4:

https://who.paris.inria.fr/Gaetan.Leurent/files/MD4_FSE08.pd...

It takes 2^102 hashes to brute force this for MD4, which is weaker than MD5. A bitcoin Antminer K7 will set you back $2,000, and it gets 58 TH/s for sha256, which is slower than MD5 or MD4. Let’s ignore that MD5 is more complex than MD4, and let’s say conservatively that similar hardware might be twice as fast for MD5 (SHA256 is really only 20-30% slower on a cpu). It’ll take 2^102/58e12/2/60/60/24/365, or about 1.4 billion years to do a second preimage attack with current hardware. So you could do that 3 times before the sun dies.

If you want to reduce that to 1.4 years, you could maybe buy a billion K7’s for $2 trillion. And each requires 2.8kW so you’ll need to find 2.8 terawatts somewhere. That’s 34 trillion kWh for 1.4 years. US yearly energy consumption is 4 trillion kWh.

It will be a while, probably decades or more, before there’s a tractable second preimage attack here.

Yes, there are stronger hashes out there than MD5, but for file verification (which is what it’s being used for) it’s fine. Safe, even. The legal folks should probably switch someday, and it’ll probably be convenient to do so since many crypto libraries won’t even let you use MD5 unless you pass a “not for security” argument.

But there’s no crisis. They can take their time.

Re: The Curious Case of MD5

#63

Post by tneely »

wyldfire wrote: post #45
tneely wrote: post #43

We still see heavy use of MD5 in genomics as well. It's effectively used to generate a single identifier that can be used to reference a specific genome assembly. There have been discussions and attempts to move to other, more secure algorithms, but the community and its tooling is too deeply entrenched in using the MD5 for the reference that it would take a herculean effort to change. I'm personally of the opinion t…

> there's not really any relevant attack space. Then why use a cryptographic hash at all? much better hashes out there that only strive for distribution/avalanche. https://en.wikipedia.org/wiki/Non-cryptographic_hash_functio...

Yeah that's fair - it doesn't need to be cryptographic. But someone back in the day decided MD5 was what they wanted and it stuck. It always raises alarms with pen testers and security scans at work, and each time we have to explain that the cryptographic security is irrelevant; it's just some unfortunate genomics standard we need to support.

Re: The Curious Case of MD5

#64

Post by bawolff »

I've always wondered what would happen if some black hat made all their payloads have the same hash in the hopes that if it ever went to court the confusion would hinder the proceedings.

Like i get that md5 is essentially a unique identifier and not meant to protect against malicious interference but if all the exhibits had the same identifier surely that would confuse people.

Re: The Curious Case of MD5

#65

Post by jeffparsons »

danpalmer wrote: post #6
kozak wrote: post #2

I still use MD5 as a 128-bit checksum algorithm that is fast and universally supported and compatible everywhere. In this role it's still useful, just don't expect it to be a cryptographic hash anymore.

While I see the point, what starts as a checksum can easily become relied upon for security over time, after all, checking whether bits have been modified accidentally on purpose, is a subtle distinction in many systems. SHA256 is also near universally supported and doesn’t have this drawback. The only cases where MD5 would be available and SHA256 wouldn’t, is systems that are out of security support anyway, where th…

This happened to me. Users initially couldn't directly control the content being hashed, because it contained a random element (via UUID). Later, the API surface expanded.

Luckily, my personal rule is to default to a cryptographic hash unless I can convince myself that cryptographic robustness will never matter and performance definitely will matter, rather than the other way around.

In this specific case all users were internal to the company, so it wouldn't have really mattered if it was vulnerable. But it could just have easily been an external user-facing thing.

Re: The Curious Case of MD5

#67

Post by StillBored »

Sigh, been having this conversation in a related codebase. Md5 is just as fine as any other generic hash function if its being used as a non-unique key, which for many cases replacing it with one of the more "secure" alternatives does nothing except for the fact that the resulting hashes are frequently longer, thereby further reducing the statistical chance of an accidental collision. For something like a document store, duplication system, etc, simply taking the extra step of doing a binary comparison against the text associated with the hash assures that accidental (or intentional) collisions are handled. With the bonus that you probably get to either publish a paper or detect someone trying to attack the system should the text comparison fail.

And given the history of cryptographic hashes, i'm even more convinced that anyone depending on sha3/whatever being better than md5/etc over the next 10-20 years is fooling themselves.

Now would I use it in a secure boot chain/etc as a stamp of uniqueness? Probably not.

Re: The Curious Case of MD5

#68

Post by CaliforniaKarl »

I've been wondering, is there a term for a type of attack like this:

Given a message M, length function L(), and MD5 hash function H(); is there an attack which can generate message M', such that H(M)==H(M') _and_ L(M)==L(M')?

In other words: Two different messages, both of the same length, with the same hash?

It's almost like a chosen prefix collision attack, but with no prefix (so P is empty) and a given message (M is known, M' is up to the attacker).

I ask because I frequently use GridFTP for data transfer, and it uses both the file length and the MD5 has to verify that files were transferred correctly.

Re: The Curious Case of MD5

#69

Post by supriyo-biswas »

CaliforniaKarl wrote: post #68

I've been wondering, is there a term for a type of attack like this: Given a message M, length function L(), and MD5 hash function H(); is there an attack which can generate message M', such that H(M)==H(M') _and_ L(M)==L(M')? In other words: Two different messages, both of the same length, with the same hash? It's almost like a chosen prefix collision attack, but with no prefix (so P is empty) and a given message (M…

That is still an attack on the second preimage or a collision resistance properties of the hash function. Most collisions do work this way, for example see [1].

[1] https://github.com/corkami/collisions

Re: The Curious Case of MD5

#70

Post by abhibeckert »

CaliforniaKarl wrote: post #68

I've been wondering, is there a term for a type of attack like this: Given a message M, length function L(), and MD5 hash function H(); is there an attack which can generate message M', such that H(M)==H(M') _and_ L(M)==L(M')? In other words: Two different messages, both of the same length, with the same hash? It's almost like a chosen prefix collision attack, but with no prefix (so P is empty) and a given message (M…

I don't know anything about GridFTP - but there's a huge difference between verifying if files were "transferred correctly" and verifying that files were transferred without being tampered with by a malicious party.

MD5 is fine for the first task, and totally unacceptable for the second.

Post reply on HN