Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

111–120 of 416 posts

Re: The largest Git repo

#111

Post by alkonaut »

taylorlafrinere wrote: post #60
kk1274 wrote: post #53

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

Most of that 300GB isn't text. There are test assets, images, videos, built binaries, vhd's, etc. Also, I should be clear that that 300GB is just at tip (no history). We can debate about whether or not those things should be checked into the repo but they are there now.

How did you go about creating the central repo and how long did it take? A 2Gb at tip svn repo with 100k commits is taking me many days and each odd failure typically has me restart the process after filtering out some obscure part of the tree.

Edit: read in another comment that you dropped the history. Understandable, but can appreciate how that would add to friction (devs having to look through two different histories).

Re: The largest Git repo

#112

Post by Analemma_ »

cosinetau wrote: post #99
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

This was a very interesting point. It sounds like there are some serious architectural limitations on Windows, and this makes me believe the same might be true for the NT kernel, and that MS might not be interested in doing heavy refactoring of it. I'm not a frequent Windows user, or a Windows dev at all. Does anyone know of any consequences that MS's decision might mean, if this hypothesis is true?

It seems presumptive to say that "If you can't use multiple repos, your architecture must be bad". I could just as easily counter with, "If you're using multiple repos, it must mean you have an unnecessarily complex and fragile microservice architecture"

Re: The largest Git repo

#114

Post by maxxxxx »

eyq1 wrote: post #106
maxxxxx wrote: post #73

Earlier quoted context omitted.

Here is my cynical view: From what I know they have a history of not using their own tools. They didn't use SourceSafe, but Perforce. Then they made an effort to switch to TFS, realized that it sucks and moved on to git. You can see the same pattern in Windows desktop apps. They didn't use MFC for themselves, didn't use Winforms, used WPF only a little.

If they didn't use MFC, WinForms and WPF what exactly were they using? Flash?

Win32

Re: The largest Git repo

#115

Post by electrum »

vtbassmatt wrote: post #25
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

Facebook cited similar reasons for having a single large repository: https://code.facebook.com/posts/218678814984400/scaling-merc...

Re: The largest Git repo

#116

Post by puzzle »

bpicolo wrote: post #64
Jyaif wrote: post #51

Earlier quoted context omitted.

pros to big repo: -dont have to spend time to think about defining interfaces cons: -history is full of crap you dont care about -tests take forever to run -tooling breaks down completely, though thanks to MS the limit was increased seriously

Are the big monorepo companies actually waiting for global test suite completion for every change? I'd doubt that, I'm sure they're using intelligent tools to figure out what tests to actually run. Compute for testing is massively expensive at that scale so it's an obvious place to optimize

Google's build and testing system is smart in which tests to run, as you suspect, but it still has a very, very large footprint.

Re: The largest Git repo

#117

Post by afpx »

vtbassmatt wrote: post #25
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

That's a really interesting article. I wish I would have found it before going down a similar path with my team, recently.

These types of use cases seem so commonly encountered that there should be a list of best practices in the Git docs.

Re: The largest Git repo

#119

Post by kodablah »

StavrosK wrote: post #102
sp332 wrote: post #17

Archive Team is making a distributed backup of the Internet Archive. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK Currently the method getting the most attention is to put the data into git-annex repos, and then have clients just download as many files as they have storage space for. But because of limitations with git, each repo can only handle about 100,000 files even if they are not "hydrated". http:…

Wouldn't IPFS be much, much more suitable for this purpose?

IIRC, there are permanence and equitable sharing guarantee concerns with IPFS. The former at least can be helped by pinning I think.

Re: The largest Git repo

#120

Post by cwyers »

planteen wrote: post #104
cosinetau wrote: post #99

Earlier quoted context omitted.

This was a very interesting point. It sounds like there are some serious architectural limitations on Windows, and this makes me believe the same might be true for the NT kernel, and that MS might not be interested in doing heavy refactoring of it. I'm not a frequent Windows user, or a Windows dev at all. Does anyone know of any consequences that MS's decision might mean, if this hypothesis is true?

Google also uses a single giant repo...

And Facebook.
Post reply on HN