Supercharging the Git Commit Graph IV: Bloom Filters
blogs.msdn.microsoft.com
Supercharging the Git Commit Graph IV: Bloom Filters
1–10 of 44 posts
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#2This is interesting work.
I've never noticed the referenced git operation as being slow. In my experience,
git log --
Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals.
I love seeing bloom filters in use in widely-used software in real life, in any case!
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#3Does it work when I'm looking for all commits changing a directory? I.e. Do directories get added to the bloom filter or just filenames?
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#4This is interesting work. I've never noticed the referenced git operation as being slow. In my experience, git log -- has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals. I love seeing bloom fi…
>10s of thousands of commits
Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that.
You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits.
Microsoft even created a virtual filesystem which they run git on top of: https://github.com/Microsoft/gvfs
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#5Does it work when I'm looking for all commits changing a directory? I.e. Do directories get added to the bloom filter or just filenames?
[deleted]
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#6Does it work when I'm looking for all commits changing a directory? I.e. Do directories get added to the bloom filter or just filenames?
I thought git didn't have a concept of directories, just blobs that have a path? If so, I presume "all commits changing a directory" is found by individually checking each file contained by the directory.
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#7This is interesting work. I've never noticed the referenced git operation as being slow. In my experience, git log -- has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals. I love seeing bloom fi…
>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of:…
Well, Google has succesfully defended their monorepo approach on multiple occasions.
So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do.
[1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
[2] On monolithic repositories (2014) https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repos...
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#8Earlier quoted context omitted.
>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of:…
Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/…
Odd that people are including Twitter as a data point in the monorepo debate. I was there when they transitioned to a Git monorepo, and it nearly killed the companies productivity.
Imagine having to torrent a starter pack of the repo, then trying to sync and failing multiple times. Then after it syncs, it could take minutes to do common operations such as change a branch, or check in a file. Not to mention many developer laptops had to have their SSD upgraded as they didn't have enough space to get the whole thing and do work.
I'm sure they've improved it since then, but this was one of many things Twitter simply cargo culted from Google without any real benefit.
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#9Earlier quoted context omitted.
>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of:…
Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/…
> So, before dissing this approach as 'non-SV-canon',
what's that supposed to mean and how do you get it from the parent comment?
Re: Supercharging the Git Commit Graph IV: Bloom Filters
#10Earlier quoted context omitted.
>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of:…
Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/…
Wait a sec, I thought the monorepo was the SV canon. As in, "we have to keep our velocity high, to deliver customer value multiple times a day, so everyone should be able to make changes across the whole stack in a single commit, and release to production in five minutes."
In my opinion it boils down to a single factor: if you don't have proper APIs between your components, then you will need to make cross-cutting changes across the whole codebase on a daily basis, thus you need everything in the same repo to work effectively. If you don't have the time to work out the proper boundaries/APIs between your components, then sure, by all means go for the monorepo! But that's like not fixing a blown tire just because you would need to stop to do it.