Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A new hash algorithm for Git

lwn.net

Post reply on HN
Print view View original

191–200 of 240 posts

Re: A new hash algorithm for Git

#191

Post by wyoung2 »

seniorsassycat wrote: post #185
wyoung2 wrote: post #164

Earlier quoted context omitted.

> Furthermore, the evil.c file with the same SHA1 hash would need to be valid C code that does something evil while still yielding the same hash ...and also produce an innocent-looking diff! I mean, you could stuff a bunch of random bytes into a C comment to force the desired hash in the output using these documented attack techniques, but anyone inspecting the diffs between versions is likely to see such an explosio…

That's an argument for why you shouldn't worry about sha1 attacks in source control, but we should take the attack for granted when discussing how to mitigate the attack. If we weren't worried about sha1 collisions in git then we wouldn't switch to a new hash function.

When is the right time to worry? Maybe wait until someone publishes a practical attack, then wait years for the new code to get sufficiently far out into the world that you can switch to it?

I mean, I see you're expressing concern, but the first major red flag on this went up three years ago, and another big one went up last month. (https://sha-mbles.github.io/)

When we dealt with this same problem over in Fossil land, we ended up needing to wait most of three years for Debian to finally ship a new enough binary that we could switch the default to SHA-3. Fortunately (?) RHEL doesn't ship Fossil, else we'd likely have had to wait even longer.

Atop that same problem, Git's also got tremendously more inertia. Git has to wait out not only the Debian and RHEL stable package policies but also all of that infrastructure tooling they brag on. Every random programmer's editor, merge tool, Git front end... all of that which a project depends on will have to convert over before that one project can move to a post-SHA-1 future.

This is going to be a colossal mess.

Re: A new hash algorithm for Git

#192

Post by SQLite »

_verandaguy wrote: post #119
strenholme wrote: post #97

I’m already seeing a lot of discussion both here and over at LWN about which hash algorithm to use. The Git team made the right choice: SHA2-256 is the best choice here; it has been around for 19 years and is still secure, in the sense that there are no known attacks against it. Both BLAKE[2/3] and SHA-3 (Keccak) have been around for 12 years and are both secure; just as BLAKE2 and BLAKE3 are faster reduced round var…

Honest question: what are the use cases in Git where hash computation speed is a meaningful optimization?

My experience in developing and maintaining Fossil is that the hashing speed is not a factor, unless you are checking in huge JPEGs or MP3s or something. And even then, the relative performance of the various hash algorithms is not enough to worry about.

Re: A new hash algorithm for Git

#193

Post by SQLite »

seniorsassycat wrote: post #183
ajkjk wrote: post #180

Earlier quoted context omitted.

> applications of blockchain technology Are we saying this now? More like blockchain is an application of git technology.

https://en.wikipedia.org/wiki/Merkle_tree

"blockchain" is self-descriptive, easier to pronounce (only two syllables instead of three), and easier to spell correctly. :-)

Re: A new hash algorithm for Git

#194

Post by masklinn »

gregmac wrote: post #112
wongarsu wrote: post #81

Earlier quoted context omitted.

You can't change past commits to add that hash (without changing all commit hashes), so this method could only protect new commits. For any existing repo this would lead to a very weird security model: We admit that sha1 hashes are broken, and only guarantee that commits made by git versions newer than git x.x.x are safe from after-the-fact modification (or alternatively only commits made after date X).

My inclination is that protecting only new commits might be enough, but it gets me thinking: What would a practical attack on this look like, assuming sha1 was broken? Let's say I'm trying to insert a line of code that does something nefarious, and that it's now trivial to generate "magic text" you can stick anywhere in a file (eg, inside a comment at the end of a line) to get any desired sha1 hash. Are all the other…

> Are all the other future commits still valid, or am I going to suddenly get conflicts or garbled text? Depending on where the modification is done, that code might have gone through much more churn -- especially if there are a bunch of sha-256 commits after it (which I can't attack). I don't know enough about how git stores content blobs to answer this.

A blob is a "snapshot" of a file. The next version of a file is a completely different blob with no direct relation to the previous.

"Pack files" use delta compression in order to lower the actual size of "similar" blobs.

You could get conflicts if you tried merging or rebasing over the nefarious blob, and the "patch history" (git log -p, which builds the patch view on the fly) would show possibly unexpected complete file replacements.

Re: A new hash algorithm for Git

#195

Post by wyoung2 »

danShumway wrote: post #182
wyoung2 wrote: post #163

Earlier quoted context omitted.

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like: > Rebasing is the same as lying And I think, "Holy crud do I not want to be part of this community." The nice thing about Git is that (within reason) once I understood it, I was able to use…

> sometimes I don't care about history and I'm just trying to coordinate developers across timelines

The fact that Fossil preserves history does not prevent you from coordinating with people across timelines. It is rather the whole point of a DVCS.

> conflating a workflow decision with a moral failing

I think it's fairer to say that we don't think a data repository is any place for lies of any sort, even white lies.

> I've learned to be somewhat skeptical of programming/workflow heuristics advertised as rules, and to be very skeptical of heuristics advertised as ideologies.

Sure, flexible tools are often better than inflexible ones, but you also have to consider the cost of the flexibility. Here, it means someone can say "this happened at some point in the past," and it's just plain wrong.

That isn't always an important thing. Most filesystems and databases operate on the same principle, presenting only the current truth, not any past truth.

Yet, we also have snapshotting in DBMSes and filesystems, because it's often very useful to be able to say, "This was the state of the system as of 2020.02.04."

You don't need a snapshotting filesystem for everything, and you don't need Fossil for everything, but it sure is nice to have ready access to both when needed.

> You've never accidentally committed a password to repo, or had to respond to a takedown request?

Fossil has shunning for that: https://fossil-scm.org/fossil/doc/trunk/www/shunning.wiki

And no, shunning is nothing at all like rebase, which should be clear from the article.

Fossil also has the `amend` command: http://fossil-scm.org/fossil/help?cmd=amend

And no, it is also not like rebase, because it only adds to the project history, it never destroys information.

Re: A new hash algorithm for Git

#196

Post by wyoung2 »

kazinator wrote: post #186
wyoung2 wrote: post #163

Earlier quoted context omitted.

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

The diatribe against rebasing is stupid. In fact, not having more than one parent is a good thing because you with multiple parents, you don't know what is relevant. The history has turned into a hairball. When you try to navigate back in time, you face forking roads at every step and it turns into a maze walk. The point is valid that when we rebase, we are losing history: the context of where that change was origina…

> history does not matter if the change was parented in some temporary context

It does if it means a big ball o' hackage lands on the public working branch, since it complicates merges, backouts, cherrypicks, and bisects.

Git users can also hide individual commit messages behind one big combined message, losing part of the project's development history and logical progression.

When I pull your repo and build it, and I find that it doesn't build on my system, I don't want to dig through a 500-line merge commit to figure out why you changed this one line from the one that used to build last week, I want the 14-line diff it was part of so I can begin to understand what you were thinking when you committed it. If I later find out that that 14-line change was wrong but the rest of your 500-line merge was fine, I want to be able to back it out with a single command. (In Fossil, it's `fossil merge --backout abcd1234`.)

> confuse other people with irrelevant information when they try to navigate the history.

How much time do you spend navigating the project's history vs looking at the tip of the current branch?

I'd wager that the times you dig back into the history, it's because you are in fact trying to figure out why you got here, which means a trail of detailed breadcrumbs will be more likely helpful than "...and between one week and the next, something changed in commit abcd1234, but we've lost all of its internal context, so we'll be spending next week reconstructing it because Angie's on vacation now."

Re: A new hash algorithm for Git

#197

Post by kazinator »

wyoung2 wrote: post #163
kazinator wrote: post #125

Earlier quoted context omitted.

That cute rhetoric will not fool anyone. Common git workflows use fairly succinct git commands: git diff git commit -p git rebase -i HEAD~3 The command quoted in my original comment is just this we strip away the SHA256 garbage: git log abac87a..f787cac (Or maybe it is: git log abac87a^..f787cac^ I cannot guess whether the ^ operator still has the same meaning or whether it is part of this ^{sha...} notation.) The ha…

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

Regarding (1), not "everything will work as before".

What happens if a Fossil repo that has had SHA3 commits written to it is accessed by old Fossil software before that change was introduced?

Re: A new hash algorithm for Git

#198

Post by kazinator »

danShumway wrote: post #182
wyoung2 wrote: post #163

Earlier quoted context omitted.

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like: > Rebasing is the same as lying And I think, "Holy crud do I not want to be part of this community." The nice thing about Git is that (within reason) once I understood it, I was able to use…

I have no interest in Fossil because it stores stuff in sqlite databases instead of the filesystem which I think is a stupid approach. I'm also not interested in version control systems that are dragging along a wiki and bug tracker. I just want a C program in /usr/bin that does version control.

Re: A new hash algorithm for Git

#199

Post by remram »

wyoung2 wrote: post #179
tjoff wrote: post #175

Earlier quoted context omitted.

Many diff tools don't highlight whitespace-only changes. Or at least not in a clear manner. Also, if something is replaced in the history how often do people go back and view diffs in old code? Hardly often enough to rely on it being spotted.

It only takes one person to raise the flag. Sure, many thousands of people doing blind "git clone && configure && sudo make install" could be burned by a problem like this, but someone would eventually do a diff and see the problem on any project big enough to have those thousands of trusting users in the first place. I'm not excusing these SHA-1 weaknesses, only pointing out that it won't be trivial to apply them to…

The problem is that we are considering an issue where different people can get different objects for the same hash. If the people checking all see the valid files, they cannot raise any alarms to save the poor victims who got poisoned with the wrong objects. They'll clone from the wrong fork, and no amount of checking hashes or signed tags will prevent them from running compromised code.

Re: A new hash algorithm for Git

#200

Post by john-radio »

robjan wrote: post #117
john-radio wrote: post #115

Earlier quoted context omitted.

OP's link says it is "subscription-only content," but it is still publicly available. It says that it has been "made available by an LWN subscriber." How does that work?

Subscribers receive a sharing link which can be used to share articles with friends. They tolerate sharing on HN because it brings in new customers.

That's one smart model.
Post reply on HN