Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supercharging the Git Commit Graph IV: Bloom Filters

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#31

Post by daveFNbuck »

edejong wrote: post #7
tzahola wrote: post #4

Earlier quoted context omitted.

>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of:…

Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/…

Google doesn't use Git, so their defense of a monorepo doesn't say anything about whether a monorepo is an abuse of Git.

Your Twitter link defends monorepos in principle but is very clear about how Git is not designed to handle monorepos well. This seems to support the thesis that a monorepo is an abuse of Git.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#33

Post by hinkley »

tzahola wrote: post #10
edejong wrote: post #7

Earlier quoted context omitted.

Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/…

Wait a sec, I thought the monorepo was the SV canon. As in, "we have to keep our velocity high, to deliver customer value multiple times a day, so everyone should be able to make changes across the whole stack in a single commit, and release to production in five minutes." In my opinion it boils down to a single factor: if you don't have proper APIs between your components, then you will need to make cross-cutting ch…

Monorepo is also an answer (though perhaps not the best) to the problem of having dependencies that are volatile.

If you can build your design philosophy on getting library code to be stable fairly quickly, then making a change to the system need not start an avalanche of rebuilds and redeployments, making monorepo less of a win.

But people like their kitchen sinks, their one stop shopping. And who has time to think about where the best place for a piece of code is. We are doing Agile! And that apparently means “no architecture” to way too many people.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#34

Post by stochastic_monk »

avar wrote: post #13
influx wrote: post #8

Earlier quoted context omitted.

Odd that people are including Twitter as a data point in the monorepo debate. I was there when they transitioned to a Git monorepo, and it nearly killed the companies productivity. Imagine having to torrent a starter pack of the repo, then trying to sync and failing multiple times. Then after it syncs, it could take minutes to do common operations such as change a branch, or check in a file. Not to mention many devel…

[Source: I contribute to Git and talked to Twitter's "git guys" at the time at Git Merge, am not affiliated with Twitter in any way] My understanding is that they didn't cargo-cult anything. They made a conscious choice at Twitter to pursue the monorepo model. One reason they mentioned is that while they have a lot of services running all over the place, those services tend to heavily use the same underlying base lib…

Why didn’t they use submodules, then? It doesn’t force the code into the same git history but makes shared code easily deployable.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#36

Post by jwilk »

stolee wrote: post #24
kanox wrote: post #22

Can you make `$ git branch` fast? Because it takes 5-10 seconds for me on linux.

Do you mean `git branch --contains` or `git branch -vv`? There are many options to `git branch` that cause Git to be very slow. The commit-graph feature in general will make these faster by reducing time spent parsing commits. You can compute a commit-graph right now if you have Git 2.18 installed: https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar... Generation numbers will make these operations much faste…

On HN: https://news.ycombinator.com/item?id=17408985

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#37

Post by therealdrag0 »

Sir_Cmpwn wrote: post #30
mwcampbell wrote: post #28

Earlier quoted context omitted.

Sounds like your team would benefit from GVFS [1]. [1]: https://github.com/Microsoft/GVFS

No sane team is running on Windows, so probably not.

Get over it. My company gives every employee the system of their choice. Some choose Mac, some PC, some Linux. Everyone works on the same stuff and is productive.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#38

Post by kingosticks »

stolee wrote: post #17
lozenge wrote: post #3

Does it work when I'm looking for all commits changing a directory? I.e. Do directories get added to the bloom filter or just filenames?

It does! The Bloom filter stores every path that changed, including paths to trees (except the root tree, which is expected to be changed by default). When you change a file, you also change every tree above it.

So presumably that eats into the 512 item limit. If I had quite a deep directory structure (10 levels doesn't sound that unreasonable) I'd only get this speedup for commits changing 51 files or less. Right?

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#39

Post by kanox »

stolee wrote: post #24
kanox wrote: post #22

Can you make `$ git branch` fast? Because it takes 5-10 seconds for me on linux.

Do you mean `git branch --contains` or `git branch -vv`? There are many options to `git branch` that cause Git to be very slow. The commit-graph feature in general will make these faster by reducing time spent parsing commits. You can compute a commit-graph right now if you have Git 2.18 installed: https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar... Generation numbers will make these operations much faste…

A long time ago I made an alias always passes -v to git branch and this is the problem: on large repos it can take a long time to compute the "ahead X, behind Y" information. It can be fixed by aliasing to some git branch --format without %(upstream:track)

Those numbers not really useful interactively when very large, for example it doesn't help to print that one of my branches is "behind 132132". Maybe git could print "ahead 7, behind 1000+" for old stale branches stuff? This way it would limit the number of commits examined.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#40

Post by voltagex_ »

Sir_Cmpwn wrote: post #30
mwcampbell wrote: post #28

Earlier quoted context omitted.

Sounds like your team would benefit from GVFS [1]. [1]: https://github.com/Microsoft/GVFS

No sane team is running on Windows, so probably not.

I respect your open source work, but bashing Windows isn't going to help anyone. You're not going to convert anyone to Linux that way and you're not really achieving anything.
Post reply on HN