Live data from Hacker News

A new hash algorithm for Git

lwn.net

201–210 of 240 posts

Re: A new hash algorithm for Git

#201

Earlier quoted context omitted.

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like: > Rebasing is the same as lying And I think, "Holy crud do I not want to be part of this community." The nice thing about Git is that (within reason) once I understood it, I was able to use…

I have no interest in Fossil because it stores stuff in sqlite databases instead of the filesystem which I think is a stupid approach. I'm also not interested in version control systems that are dragging along a wiki and bug tracker. I just want a C program in /usr/bin that does version control.

SQLite can be considerably faster than the filesystem: https://www.sqlite.org/fasterthanfs.html

If you think your filesystem-based Git repo is easy to manipulate, go poking around in there, and what you'll find is a bespoke one-off pile-of-files database! Given a choice between Git's DB and SQLite, I put more trust into SQLite.

> I just want a C program in /usr/bin that does version control.

...which Git doesn't provide. Git is hundreds of files scattered all over your filesystem, a large number of which aren't C binaries anyway, and of those that are, only one of them is the front-end program sitting in /usr/bin, whereas Fossil can be built to a single static executable in /usr/bin.

And if you can't build Fossil statically on your system, it's likely due to an OS limitation rather than something about Fossil itself, as on RHEL where they've made fully static linking rather difficult in the past few releases.

Getting back to Git, large chunks of Git are written in POSIX shell, Perl, Python, and Tcl/Tk. Almost all of Fossil is written in C, and the rest of the code is embedded within that binary running under built-in interpreters rather than depending on platform interpreters.

This has nice knock-on effects, one of which is that Fossil is truly native on Windows, whereas you have to drag along a Linux portability environment to run Git on Windows. Another is that Fossil plays nicely with chroot/jail/container technology.

> I'm also not interested in version control systems that are dragging along a wiki and bug tracker.

Not a GitHub or GitLab user, then, I'm guessing?

Re: A new hash algorithm for Git

#202

Earlier quoted context omitted.

> BLAKE is faster when using software to perform the hash Is BLAKE 3 still faster than sha-256 when using the cpu speciliazed instructions? I think most modern desktop CPUs has built-in instructions for SHA256. I’m guessing when people compare BLAKE 3 to SHA 256 they’re comparing software to software, but this wouldn’t be the case in reality?

I haven’t seen any benchmarks for BLAKE3 vs. the Intel/AMD SHA extensions. My guess is that Intel hardware accelerated SHA-256 will be faster than BLAKE3 running in software for most real world uses. I can tell you this much: It is only with Ice Lake, which was released in the last year, that mainstream Intel chips finally got native hi speed SHA-NI support. Coffee Lake and Comet Lake, which are still the CPUs in a l…

AMD Zen supports SHA extensions across all SKUs. Here are `openssl speed` numbers on an AMD EPYC 3201:

  type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
  blake2s256       46720.33k   187461.21k   305314.65k   373840.55k   398207.66k   401528.15k
  blake2b512       38423.44k   155318.81k   422325.08k   592401.75k   674843.31k   681743.70k
  sha256           84620.44k   279840.47k   723573.76k  1199678.81k  1484693.50k  1510484.65k
  sha512           33854.38k   135674.20k   275343.70k   444872.36k   545802.92k   554166.95k
  sha3-256         26146.35k   103860.27k   253944.92k   308119.21k   347477.33k   351906.47k
  sha3-512         26349.83k   105590.85k   144236.03k   173082.62k   189448.19k   189814.10k
It's possible that Blake3 might be faster than accelerated SHA-256 on large inputs, where Blake3 can maximally leverage its SIMD friendliness. OTOH, Blake3 really pushes the envelope in terms of minimal security margin. Performance isn't everything. SHA-3 is so slow because NIST wanted a failsafe.

OpenSSL info:

  OpenSSL 1.1.1c  28 May 2019
  built on: Tue Aug 20 11:46:33 2019 UTC
  options:bn(64,64) rc4(8x,int) des(int) aes(partial) blowfish(ptr) 
  compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-D7S1fy/openssl-1.1.1c=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
NOTE: /proc/cpuinfo shows sha_ni detection, and the apt-get source of this version of OpenSSL confirms SHA extension support in the source code, but I didn't confirm that it was actually being used at runtime.

Re: A new hash algorithm for Git

#203

Earlier quoted context omitted.

> That cute rhetoric will not fool anyone. Well, let's see, the Fossil equivalents are: 1. Do nothing at all for a conversion from the SHA-1 to SHA-3 — yes, 3, not 2 as in Git! — because it's automatic for months now and dead easy going back 3 years now. ( https://www.fossil-scm.org/fossil/doc/trunk/www/hashpolicy.w... ) 2. "fossil diff" 3. "fossil ci" 4. Why are you rebasing in the first place, again? https://www.fo…

Regarding (1), not "everything will work as before". What happens if a Fossil repo that has had SHA3 commits written to it is accessed by old Fossil software before that change was introduced?

If you try to use Fossil 1.37 — the last 1.x release — to clone a repo that has SHA-3 hashed artifacts in it, it says, "server returned an error - clone aborted". Since 1.37 pre-dates this feature, it can't give a more detailed diagnosis than that.

If you have an old clone made from before the transition and try to update it, I'm not sure what it says, since I don't have any of those around any more. It has, after all, been three years since Fossil began to move on this problem, so that it's largely a past issue for us now.

This transition time was indeed annoying for us over in Fossil land, but Git's going to have to go through a transition like this, too. The question isn't whether but how long we'll have to wait for it to begin and how long it'll take to complete.

Re: A new hash algorithm for Git

#204
post #199

Earlier quoted context omitted.

It only takes one person to raise the flag. Sure, many thousands of people doing blind "git clone && configure && sudo make install" could be burned by a problem like this, but someone would eventually do a diff and see the problem on any project big enough to have those thousands of trusting users in the first place. I'm not excusing these SHA-1 weaknesses, only pointing out that it won't be trivial to apply them to…

The problem is that we are considering an issue where different people can get different objects for the same hash. If the people checking all see the valid files, they cannot raise any alarms to save the poor victims who got poisoned with the wrong objects. They'll clone from the wrong fork, and no amount of checking hashes or signed tags will prevent them from running compromised code.

> If the people checking all see the valid files

...which will likely contain thousands of bytes of pseudorandom data in order to force the hash collision...

> they cannot raise any alarms

You think a human won't be able to notice that the diff from the last version they tested looks awfully funny? Code that can fool the compiler into producing an evil binary is one thing, but code that can pass a human code review is quite another.

You might be surprised how often that occurs.

I don't do a diff before each third-party DVCS repo pull, but I do diff the code when integrating such third-party code into my projects, if only so I understand what they've done since the last time I updated. Commit messages, ChangeLogs, and release announcements only get you so far.

Back when I was producing binary packages for a popular software distribution, I'd often be forced to diff the code when producing new binaries, since several of the popular binary package distribution systems are based on patches atop pristine upstream source packages. (RPM, DEB, Cygwin packages...)

Each time a binary package creator updates, there's a good chance they've had to diff the versions to work out how to apply their old distro-specific patches atop the new codebase.

Someone's going to notice the first time this happens, and my guess is that it'll happen rather quickly.

Re: A new hash algorithm for Git

#205

Earlier quoted context omitted.

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like: > Rebasing is the same as lying And I think, "Holy crud do I not want to be part of this community." The nice thing about Git is that (within reason) once I understood it, I was able to use…

> sometimes I don't care about history and I'm just trying to coordinate developers across timelines The fact that Fossil preserves history does not prevent you from coordinating with people across timelines. It is rather the whole point of a DVCS. > conflating a workflow decision with a moral failing I think it's fairer to say that we don't think a data repository is any place for lies of any sort, even white lies.…

My understanding is that shunning is blacklisting specific artifacts. That's nice, but I don't understand how that solves the problem.

When I revise history in Git, even if it's just doing something as simple as removing sensitive information, I often need to replace that information, either through new commits, or by introducing minor edits to surrounding commits. I could add those changes on top of my current HEAD, but then checkouts of old versions would be broken. On the other hand, if I can just replay my commits while inserting extra code, I'll end up with something that's pretty close to my original history, with just the offending information excluded/replaced.

That carries the cost that people will need to force pull my repo, but at least the repo history will still roughly correspond to what development looked like, rather than being out-of-order and mostly impossible to build except for at my current HEAD.

As a followup question, what do you do if the sensitive information you need to exclude is in a commit message? `amend` won't help you, since it's not destroying information. Do you shun that commit and then... what?

It just seems like destroying information isn't enough unless you can also replace it?

> Sure, flexible tools are often better than inflexible ones, but you also have to consider the cost of the flexibility.

I appreciate this -- I like having multiple tools for different purposes. I don't see a problem with having a VC that focuses on auditability, or having one that goes in a radically different direction from Git. Fossil has very interesting ideas, which is why I try to pay it some attention whenever I see it mentioned or linked to.

However, whenever I follow those links and start digging deeper into the philosophy behind its design decisions, inevitably the conversation changes from, "here's our alternative approach to Git" to "what Git does is fundamentally wrong". It's not, "Fossil doesn't have this problem because we eschew rebasing", it's "why would anyone rebase?"

(Nearly) all architectural decisions have good and bad consequences. Sometimes those consequences are imbalanced, so we have heuristics that can say things like, "often X is a bad idea." That's fine.

More harmfully, sometimes people extend heuristics into rules that say, "it's never a good idea to do X". Programming rules are usually wrong.

But programming ideologies the worst, because they say, "there is something mentally or morally wrong with a person who would do X". This is toxic for the reasons that Fossil devs already mention in their documentation:

> programmers should avoid linking their code with their sense of self

Programming ideologies explicitly encourage developers to have egos, because ideology conflates architectural decisions and workflow processes with individual worth. Programming ideologies make it harder for people to grow as programmers, because they tie intellectual growth to fears about being wrong. They're completely toxic.

And is Fossil's documentation promoting an ideology? I'm guessing that you'd disagree with me on this, but my take is that when Fossil's official documentation says things like:

> Honorable writers adjust their narrative to fit history. Rebase adjusts history to fit the narrative.

or

> It is dishonest. It deliberately omits historical information. It causes problems for collaboration. And it has no offsetting benefits.

That's not designing a focused tool to support specific heuristics, or making a case that, "sometimes strict auditability is important". That's just trolling for fights.

Re: A new hash algorithm for Git

#206

Earlier quoted context omitted.

> sometimes I don't care about history and I'm just trying to coordinate developers across timelines The fact that Fossil preserves history does not prevent you from coordinating with people across timelines. It is rather the whole point of a DVCS. > conflating a workflow decision with a moral failing I think it's fairer to say that we don't think a data repository is any place for lies of any sort, even white lies.…

My understanding is that shunning is blacklisting specific artifacts. That's nice, but I don't understand how that solves the problem. When I revise history in Git, even if it's just doing something as simple as removing sensitive information, I often need to replace that information, either through new commits, or by introducing minor edits to surrounding commits. I could add those changes on top of my current HEAD,…

> ideology conflates architectural decisions and workflow processes with individual worth

No. You start with the ideology based on your local culture and project needs, then you pick the tool that supports your project's needs.

This is why we spend so much time talking about philosophy in the Fossil vs. Git article, particularly this section: https://fossil-scm.org/fossil/doc/trunk/www/fossil-v-git.wik...

Which of the two philosophies matches better with the way your project works? That alone is a pretty good guide to whether you want Fossil or Git. (Or something else!)

Re: A new hash algorithm for Git

#207
post #202

Earlier quoted context omitted.

I haven’t seen any benchmarks for BLAKE3 vs. the Intel/AMD SHA extensions. My guess is that Intel hardware accelerated SHA-256 will be faster than BLAKE3 running in software for most real world uses. I can tell you this much: It is only with Ice Lake, which was released in the last year, that mainstream Intel chips finally got native hi speed SHA-NI support. Coffee Lake and Comet Lake, which are still the CPUs in a l…

AMD Zen supports SHA extensions across all SKUs. Here are `openssl speed` numbers on an AMD EPYC 3201: type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes blake2s256 46720.33k 187461.21k 305314.65k 373840.55k 398207.66k 401528.15k blake2b512 38423.44k 155318.81k 422325.08k 592401.75k 674843.31k 681743.70k sha256 84620.44k 279840.47k 723573.76k 1199678.81k 1484693.50k 1510484.65k sha512 33854.38k 135674…

Assuming Blake3 will be across the board 43% faster (7 instead of 10 rounds) than 32-bit blake2s256, we would get:

  Blake3  SHA-256
   66743    84620  Tiny
  534057  1199679  Medium (1024 bytes)
  573611  1510485  Largeish (16384 bytes)
This is based on the parent’s numbers with a fudge factor to account for Blake3 being a faster version of blake2s256 (i.e. the 32-bit version of Blake2 which is the only version in Blake3)

Of course, this does take in to account that Blake3 has tree hashing and other modes which scale better to multiple cores.

(Edit: update figures; I need to scale up Blake2s256 not Blake2b512)

Re: A new hash algorithm for Git

#208

Earlier quoted context omitted.

Articles like this are eye opening to me, in a bad way. Every once in a while, I get really curious about giving Fossil a try, because it does have some legitimately cool ideas, and then I see the documentation saying things like: > Rebasing is the same as lying And I think, "Holy crud do I not want to be part of this community." The nice thing about Git is that (within reason) once I understood it, I was able to use…

> sometimes I don't care about history and I'm just trying to coordinate developers across timelines The fact that Fossil preserves history does not prevent you from coordinating with people across timelines. It is rather the whole point of a DVCS. > conflating a workflow decision with a moral failing I think it's fairer to say that we don't think a data repository is any place for lies of any sort, even white lies.…

> I think it's fairer to say that we don't think a data repository is any place for lies of any sort, even white lies.

I, too, wish this extreme hyperbole would be just left out of the discussion completely. It is offputting, and I think it's intentionally a bad faith argument, it fails to acknowledge the utility, the design intent, and the context behind rebase, which has been talked about at length by Linus and others.

When rebase is used as designed, according to the golden rule, it's not modifying published history, so it's not "lying". Whether rebase has safety problems is a separate issue from whether it's use as designed amounts to being "dishonest".

I'm all in favor of improved design choices, and if Fossil is making those better design choices, let them stand on their own without intentionally denigrating git and every user of git through utter exaggeration.

Re: A new hash algorithm for Git

#209

Earlier quoted context omitted.

Note the qualifier "for a Git user interface". The average git command is along the lines of "git ph-nglui --mglw=nafh Cthulhu...R'lyeh -- wgah^nagl fhtagn"

That cute rhetoric will not fool anyone. Common git workflows use fairly succinct git commands: git diff git commit -p git rebase -i HEAD~3 The command quoted in my original comment is just this we strip away the SHA256 garbage: git log abac87a..f787cac (Or maybe it is: git log abac87a^..f787cac^ I cannot guess whether the ^ operator still has the same meaning or whether it is part of this ^{sha...} notation.) The ha…

As others have pointed out, there already is precedent for ^{...}, so if you're comfortable with the other uses, I'm not sure why you should NOT be comfortable with this new addition.

Re: A new hash algorithm for Git

#210
post #161

Earlier quoted context omitted.

The way I read the fossil's authors comments, old commits continue to use sha1 hashes. A repository will be vulnerable to sha1 collision attacks as long as there is an object in the repository that has not been hashed with the new algorithm. For example, floppy.c could be replaced in a repo with file with the same sha1 hash as long as the last commit that modifies floppy.c used a sha1 hash. Right?

Just to be clear: Every time you modify a file, the new changes get put in using SHA3. In an older repository, any given commit might have some files identified using SHA1 (assuming they have not changed in 3 years) and others identified using SHA3. For example, the manifest of the latest SQLite check-in is see at ( https://www.sqlite.org/src/artifact/29a969d6b1709b80 ). You can see that most of the files have longer…

Doesn't all of this apply to git just as well, except for the last bit about the MD5 hash?

It just seems to me that the Fossil maintainers have decided that keeping all old SHA1 hashes is acceptable, while the git maintainers have decided that it is not.

Unless I've misunderstood, this is why it was "so easy" for Fossil to transition to a new hashing algorithm. Not some superiority in the design of Fossil, as implied on the Fossil forums.

Post reply on HN