Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Speed Hashing

codinghorror.com

Post reply on HN
Print view View original

11–20 of 133 posts

Re: Speed Hashing

#11

Post by codinghorror »

masklinn wrote: post #10
JoachimSchipper wrote: post #4

Hashes are designed to be fast. Password hashes (MD5crypt / SHA1crypt / bcrypt / scrypt / PBDKDF2) are designed to be slow to make dictionary attacks harder, but the hashes used in SSL and the like are designed to be as fast as possible without sacrificing too much security.

As often, he plays fast and loose with terminology and uses "checksums" for "hash functions" and "hashes" for " cryptographic hash functions". > the hashes used in SSL and the like are designed to be as fast as possible without sacrificing too much security. See also: map and set hashes. There, you're looking for speed and good distribution across the buckets, a slower hash function is not something to look forward t…

I guess, but what's the value of a "checksum" that fails to detect certain "cryptographically secure" changes in a file? MD5 is clearly just a checksum now, since it's not secure, but it was designed to be originally. And would you ever want to use a checksum that others could manipulate at will? Curious.

I feel like the line between "this is a checksum" and "this is a secure hash" is kind of illusory, other than for pure performance reasons.

Re: Speed Hashing

#12

Post by codinghorror »

exDM69 wrote: post #6

Nitpick: this statement is not exactly true: "Hashes are designed to be tamper-proof". This only applies to cryptographic hash functions, like SHA-1 and Skein. There are non-cryptogaphic hash functions which are not designed to be secure, like Dan Bernstein's DJB-family of hashes (DJBX33A and DJBX33X), that are used e.g. for hash tables with string keys.

I suppose I was thinking of Java and .NET where you don't generally get the chance to "pick" a hash function unless you're in the Crypto namespaces. There are certainly hash-based data structures like HashTables and the like, but the underlying hash algorithms are not exposed in any way, they're just magic.

Re: Speed Hashing

#13

Post by JoachimSchipper »

codinghorror wrote: post #11
masklinn wrote: post #10

Earlier quoted context omitted.

As often, he plays fast and loose with terminology and uses "checksums" for "hash functions" and "hashes" for " cryptographic hash functions". > the hashes used in SSL and the like are designed to be as fast as possible without sacrificing too much security. See also: map and set hashes. There, you're looking for speed and good distribution across the buckets, a slower hash function is not something to look forward t…

I guess, but what's the value of a "checksum" that fails to detect certain "cryptographically secure" changes in a file? MD5 is clearly just a checksum now, since it's not secure, but it was designed to be originally. And would you ever want to use a checksum that others could manipulate at will? Curious. I feel like the line between "this is a checksum" and "this is a secure hash" is kind of illusory, other than for…

E.g. ZFS uses non-cryptographical hashes to detect bitrot. TCP and IPv4 use a non-cryptographical hash to detect packet corruption. Etc.

Re: Speed Hashing

#14

Post by marcusf »

codinghorror wrote: post #12
exDM69 wrote: post #6

Nitpick: this statement is not exactly true: "Hashes are designed to be tamper-proof". This only applies to cryptographic hash functions, like SHA-1 and Skein. There are non-cryptogaphic hash functions which are not designed to be secure, like Dan Bernstein's DJB-family of hashes (DJBX33A and DJBX33X), that are used e.g. for hash tables with string keys.

I suppose I was thinking of Java and .NET where you don't generally get the chance to "pick" a hash function unless you're in the Crypto namespaces. There are certainly hash-based data structures like HashTables and the like, but the underlying hash algorithms are not exposed in any way, they're just magic.

Not to pile on with the nitpicking, but for just the cases everybody here is talking about (eg non-crypto hashing) you get to "pick" exactly the hash function. You're forced, actually, to implement hashCode() in Java if you want to be able to put the object in a HashMap, HashSet, etc.

Re: Speed Hashing

#15

Post by arethuza »

codinghorror wrote: post #12
exDM69 wrote: post #6

Nitpick: this statement is not exactly true: "Hashes are designed to be tamper-proof". This only applies to cryptographic hash functions, like SHA-1 and Skein. There are non-cryptogaphic hash functions which are not designed to be secure, like Dan Bernstein's DJB-family of hashes (DJBX33A and DJBX33X), that are used e.g. for hash tables with string keys.

I suppose I was thinking of Java and .NET where you don't generally get the chance to "pick" a hash function unless you're in the Crypto namespaces. There are certainly hash-based data structures like HashTables and the like, but the underlying hash algorithms are not exposed in any way, they're just magic.

You can override the default hash producing methods in both Java (hashCode) and C# (GetHashCode) - so while the standard hash generators might not be exposed directly they can easily be replaced.

Re: Speed Hashing

#16

Post by canop_fr »

codinghorror wrote: post #12
exDM69 wrote: post #6

Nitpick: this statement is not exactly true: "Hashes are designed to be tamper-proof". This only applies to cryptographic hash functions, like SHA-1 and Skein. There are non-cryptogaphic hash functions which are not designed to be secure, like Dan Bernstein's DJB-family of hashes (DJBX33A and DJBX33X), that are used e.g. for hash tables with string keys.

I suppose I was thinking of Java and .NET where you don't generally get the chance to "pick" a hash function unless you're in the Crypto namespaces. There are certainly hash-based data structures like HashTables and the like, but the underlying hash algorithms are not exposed in any way, they're just magic.

Using chosen hash functions in your applications isn't so rare even outside cryptography field. For example I often use hash for storage addressing (ala git). For this kind of thing you need something fast and with practically no collision (which isn't a requirement for a common hash table).

Re: Speed Hashing

#17

Post by codinghorror »

JoachimSchipper wrote: post #13
codinghorror wrote: post #11

Earlier quoted context omitted.

I guess, but what's the value of a "checksum" that fails to detect certain "cryptographically secure" changes in a file? MD5 is clearly just a checksum now, since it's not secure, but it was designed to be originally. And would you ever want to use a checksum that others could manipulate at will? Curious. I feel like the line between "this is a checksum" and "this is a secure hash" is kind of illusory, other than for…

E.g. ZFS uses non-cryptographical hashes to detect bitrot. TCP and IPv4 use a non-cryptographical hash to detect packet corruption. Etc.

terminology issue, I guess, I'd call those checksums -- where speed is the overriding concern and you're not worried about attackers changing the data underneath you. (And you actually need to uniquely identify the data in a reliable way..)

Re: Speed Hashing

#18

Post by 16s »

For long, random passwords that can be easily remembered, try SHA1_Pass. It's free, with source code and runs on Windows, Linux and Macs.

Re: Speed Hashing

#19

Post by codinghorror »

judofyr wrote: post #9

Nitpicks. It seems that he's talking about password hashing: > "Hashes are designed to be tamper-proof". Wrong. Cryptographically secure hash functions are. > "Hashes, when used for security, need to be slow." Wrong. Password hashes needs this; not SHA1/MD5 etc.

Well, I'd say a hash that has no need whatsoever to be tamper-proof (no attackers, ever) and cares only about speed is a checksum -- so maybe a terminology issue.

I agree there are certainly other uses for hashes, just trying to distinguish between hashes and checksums.

Re-reading what I wrote, I open with "Hashes are a bit like fingerprints for data. A given hash uniquely represents a file, or any arbitrary collection of data" so the context of this article is hash functions that are able to uniquely identify something in a reliable, trustworthy way -- either checksums (fast, no need for security) or hashing (slower, more reliable, possibly "secure" for some definition of secure), but always in the context of "can I trust this value to tell me that the data is really what I think it is?"

Of course there is a tradeoff with speed; a person's name can be good enough identifier (checksum) in some circumstances, but maybe other circumstances require more reliability like DNA or fingerprints (secure-ish hash), at the cost of being far slower to collect and validate and way more onerous.

Re: Speed Hashing

#20

Post by canop_fr »

marcusf wrote: post #14
codinghorror wrote: post #12

Earlier quoted context omitted.

I suppose I was thinking of Java and .NET where you don't generally get the chance to "pick" a hash function unless you're in the Crypto namespaces. There are certainly hash-based data structures like HashTables and the like, but the underlying hash algorithms are not exposed in any way, they're just magic.

Not to pile on with the nitpicking, but for just the cases everybody here is talking about (eg non-crypto hashing) you get to "pick" exactly the hash function. You're forced, actually, to implement hashCode() in Java if you want to be able to put the object in a HashMap, HashSet, etc.

You're right but that's still in a very small subset of possible hashing functions because you must produce a 4 bytes hash, which is then hashed again (to the length of the storage array). Of course you don't have as requirement to avoid collision, just to distribute as equally as possible.
Post reply on HN