Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A new hash algorithm for Git

lwn.net

Post reply on HN
Print view View original

181–190 of 240 posts

Re: A new hash algorithm for Git

#181

Post by seniorsassycat »

SQLite wrote: post #161
seniorsassycat wrote: post #143

Earlier quoted context omitted.

The way I read the fossil's authors comments, old commits continue to use sha1 hashes. A repository will be vulnerable to sha1 collision attacks as long as there is an object in the repository that has not been hashed with the new algorithm. For example, floppy.c could be replaced in a repo with file with the same sha1 hash as long as the last commit that modifies floppy.c used a sha1 hash. Right?

Just to be clear: Every time you modify a file, the new changes get put in using SHA3. In an older repository, any given commit might have some files identified using SHA1 (assuming they have not changed in 3 years) and others identified using SHA3. For example, the manifest of the latest SQLite check-in is see at ( https://www.sqlite.org/src/artifact/29a969d6b1709b80 ). You can see that most of the files have longer…

Isn't this the same attack given as an example why git is migrating hash functions in the subject article?

The attack may be difficult and unlikely I'm not questioning that, but if I understand correctly then Fossil's migration is straightforward because they did not address the same issues Git chose to.

Re: A new hash algorithm for Git

#182

Post by danShumway »

wyoung2 wrote: post #163
kazinator wrote: post #125

Earlier quoted context omitted.

That cute rhetoric will not fool anyone. Common git workflows use fairly succinct git commands: git diff git commit -p git rebase -i HEAD~3 The command quoted in my original comment is just this we strip away the SHA256 garbage: git log abac87a..f787cac (Or maybe it is: git log abac87a^..f787cac^ I cannot guess whether the ^ operator still has the same meaning or whether it is part of this ^{sha...} notation.) The ha…

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like:

> Rebasing is the same as lying

And I think, "Holy crud do I not want to be part of this community."

The nice thing about Git is that (within reason) once I understood it, I was able to use it in very flexible ways.

It's really common for different projects I manage to range all over the place from the extreme "commits as literal history" perspective all the way to the "commits as literature/guide" perspective. Sometimes I don't rebase at all, sometimes I rebase a lot. Sometimes I commit everything, all the time, sometimes I refuse to commit any code that isn't a deployable feature. Sometimes I leave branches as historical artifacts, sometimes I don't care about history and I'm just trying to coordinate developers across timelines.

That's not to say that Git isn't opinionated about some things -- nearly all good tools have at least a few strong opinions. But Git passes the (IMO extremely low) bar of not conflating a workflow decision with a moral failing. Over the years as a software engineer, I've learned to be somewhat skeptical of programming/workflow heuristics advertised as rules, and to be very skeptical of heuristics advertised as ideologies.

I really don't understand the perspective of someone who can't think of even one good reason why they would ever want to edit history. You've never accidentally committed a password to repo, or had to respond to a takedown request?

Re: A new hash algorithm for Git

#183

Post by seniorsassycat »

ajkjk wrote: post #180
wyoung2 wrote: post #165

Earlier quoted context omitted.

In addition to D. Richard Hipp's thoughts as HN user SQLite — author also of Fossil, so he oughtta know — I offer these: 1. Keep in mind that Fossil and Git are both applications of blockchain technology, which in this particular practical case means you must not only forge a single artifact's hash, you must also do it in a way that allows it to fit into the overall blockchain. 2. Fossil's sync protocol purposefully…

> applications of blockchain technology Are we saying this now? More like blockchain is an application of git technology.

https://en.wikipedia.org/wiki/Merkle_tree

Re: A new hash algorithm for Git

#184

Post by RichardCA »

velcrovan wrote: post #101
alkonaut wrote: post #95

I didn't get the argument against just converting? Sure some code bases are large and spread out, but any git repo needs to have one blessed central point, and everyone needs to be able to just re-clone from the central repository whenever history is rewritten for whatever reason (could be that a huge file is trimmed from the past etc). Why can't all commits in the Kernel history be rewritten to SHA256? (Other than t…

The whole point of git is that there doesn't need to be a blessed central point.

Most development shops are using the traditional client-server model, or self-host using Gitlab.

I personally would never allow a repo with two hashing algorithms to exist on my watch.

If you have ever had to use a tool like BFG to prune large objects from a repo you'll see it's not that bad, but it does require users to re-clone.

I would want to use the same process for SHA256 - that is let it be the default for new projects and then convert older projects based on need.

But there needs to be a BFG style conversion tool that spits out an object id map as output.

Here's more info on BFG: https://rtyley.github.io/bfg-repo-cleaner/

Re: A new hash algorithm for Git

#185

Post by seniorsassycat »

wyoung2 wrote: post #164
SQLite wrote: post #161

Earlier quoted context omitted.

Just to be clear: Every time you modify a file, the new changes get put in using SHA3. In an older repository, any given commit might have some files identified using SHA1 (assuming they have not changed in 3 years) and others identified using SHA3. For example, the manifest of the latest SQLite check-in is see at ( https://www.sqlite.org/src/artifact/29a969d6b1709b80 ). You can see that most of the files have longer…

> Furthermore, the evil.c file with the same SHA1 hash would need to be valid C code that does something evil while still yielding the same hash ...and also produce an innocent-looking diff! I mean, you could stuff a bunch of random bytes into a C comment to force the desired hash in the output using these documented attack techniques, but anyone inspecting the diffs between versions is likely to see such an explosio…

That's an argument for why you shouldn't worry about sha1 attacks in source control, but we should take the attack for granted when discussing how to mitigate the attack.

If we weren't worried about sha1 collisions in git then we wouldn't switch to a new hash function.

Re: A new hash algorithm for Git

#186

Post by kazinator »

wyoung2 wrote: post #163
kazinator wrote: post #125

Earlier quoted context omitted.

That cute rhetoric will not fool anyone. Common git workflows use fairly succinct git commands: git diff git commit -p git rebase -i HEAD~3 The command quoted in my original comment is just this we strip away the SHA256 garbage: git log abac87a..f787cac (Or maybe it is: git log abac87a^..f787cac^ I cannot guess whether the ^ operator still has the same meaning or whether it is part of this ^{sha...} notation.) The ha…

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

The diatribe against rebasing is stupid. In fact, not having more than one parent is a good thing because you with multiple parents, you don't know what is relevant. The history has turned into a hairball. When you try to navigate back in time, you face forking roads at every step and it turns into a maze walk.

The point is valid that when we rebase, we are losing history: the context of where that change was originally parented.

However, (1) the history does not matter if the change was parented in some temporary context, like your unpublished changes and (2) the information can be tracked in other ways, such as a Gerrit Change-Id (or something like it) in the commit message.

Regarding (1) the extra parent pointers in a merge commit cause retention of garbage. If we do everything with merge instead of rebase, we will never lose any of the temporary commits. If we prepare an unpublished change through numerous rebase operations, all that temporary crap will stay referenced from the head, waste space and confuse other people with irrelevant information when they try to navigate the history.

Re: A new hash algorithm for Git

#187

Post by _jal »

ajkjk wrote: post #180
wyoung2 wrote: post #165

Earlier quoted context omitted.

In addition to D. Richard Hipp's thoughts as HN user SQLite — author also of Fossil, so he oughtta know — I offer these: 1. Keep in mind that Fossil and Git are both applications of blockchain technology, which in this particular practical case means you must not only forge a single artifact's hash, you must also do it in a way that allows it to fit into the overall blockchain. 2. Fossil's sync protocol purposefully…

> applications of blockchain technology Are we saying this now? More like blockchain is an application of git technology.

No. We are not.

If you're looking for prior art, ZFS's application of Merkle trees predates both. I think there was some other public use before that, but I can't recall right now.

Re: A new hash algorithm for Git

#188

Post by SQLite »

seniorsassycat wrote: post #181
SQLite wrote: post #161

Earlier quoted context omitted.

Just to be clear: Every time you modify a file, the new changes get put in using SHA3. In an older repository, any given commit might have some files identified using SHA1 (assuming they have not changed in 3 years) and others identified using SHA3. For example, the manifest of the latest SQLite check-in is see at ( https://www.sqlite.org/src/artifact/29a969d6b1709b80 ). You can see that most of the files have longer…

Isn't this the same attack given as an example why git is migrating hash functions in the subject article? The attack may be difficult and unlikely I'm not questioning that, but if I understand correctly then Fossil's migration is straightforward because they did not address the same issues Git chose to.

> if I understand correctly then Fossil's migration is straightforward because they did not address the same issues Git chose to.

I think more is at play here.

(1) You can set Fossil to ignore all SHA1 artifacts using the "shun-sha1" hash policy.

(2) The excess complication in the Git migration strategy is likely due to the inability of the underlying Git file formats to handle two different hash algorithms in the same repository at the same time.

But, I could be wrong. Post a rebuttal if you have evidence to the contrary.

Re: A new hash algorithm for Git

#189

Post by kzrdude »

pabs3 wrote: post #12
tcharlton wrote: post #5

I can't find documentation for the command in the article: git convert-repo --to-hash=sha-256 --frobnicate-blobs --climb-subtrees \ --liability-waiver=none --use-shovels --carbon-offsets Surely some of those options aren't real...

That seems to be intended to be humour.

Well, I loved it, for one.

Re: A new hash algorithm for Git

#190

Post by minitech »

jolmg wrote: post #158
loeg wrote: post #156

Earlier quoted context omitted.

You're talking about a 160 bit truncated hash collision on SHA256, which is extraordinarily unlikely if SHA256 is not itself completely broken (moreso than SHA1 already is!). I don't think any syntax is needed for that in the porcelain CLI; it could be handled with non-user-facing commands if it ever came up (it won't).

> extraordinarily unlikely if SHA256 is not itself completely broken (moreso than SHA1 already is I was hoping I captured that by saying "very rarely". However, if SHA1 collisions can be made willingly, doesn't that mean that one can also willingly make a SHA1 hash that matches with the prefix of an existing SHA256 hash?

> doesn't that mean that one can also willingly make a SHA1 hash that matches with the prefix of an existing SHA256 hash?

No, the “prefix of an existing SHA256 hash” stops being relevant at that point – that’s just a full preimage attack on SHA1. Isn’t known to be feasible yet.

> I was hoping I captured that by saying "very rarely"

It’s rarer than that. :)

Post reply on HN