The largest Git repo • Hacker News Board

Re: The largest Git repo

#51

Post by Jyaif » Wed, May 24, 2017, 5:15 PM UTC

howinator wrote: post #14

tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

So, this is actually pretty common. I know that both Google and Facebook use a huge mono-repo for literally everything (except I think Facebook split out their Android code into a separate repo?). So, all of Facebook's and Google's code for front-end, back-end, tools, infrastructure, literally everything, lives in one repo. It's news to me that Windows decided to go that route too. Personally, I think submodules and…

pros to big repo:

-dont have to spend time to think about defining interfaces

cons:

-history is full of crap you dont care about

-tests take forever to run

-tooling breaks down completely, though thanks to MS the limit was increased seriously

Re: The largest Git repo

#52

Post by vtbassmatt » Wed, May 24, 2017, 5:16 PM UTC

sigil wrote: post #41

vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

This is amazing, congrats. I worked on Windows briefly in 2005 (the same year git was released!) and was surprised at how well Source Depot worked, especially given the sheer size of the codebase and the other SCM tools at the time. Is there anything people particularly miss about Source Depot? Something SD was good at, but git is not?

I just got a request today for an API equivalent to `sd files`, which is not something Git is natively great at without a local copy of the repo.

Re: The largest Git repo

#53

Post by kk1274 » Wed, May 24, 2017, 5:18 PM UTC

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

Re: The largest Git repo

#54

Post by gcb0 » Wed, May 24, 2017, 5:19 PM UTC

sp332 wrote: post #17

Archive Team is making a distributed backup of the Internet Archive. http://archiveteam.org/index.php?title=INTERNETARCHIVE.BAK Currently the method getting the most attention is to put the data into git-annex repos, and then have clients just download as many files as they have storage space for. But because of limitations with git, each repo can only handle about 100,000 files even if they are not "hydrated". http:…

internet archive sounds like the best ever use case for IPFS

Re: The largest Git repo

#55

Post by alkonaut » Wed, May 24, 2017, 5:19 PM UTC

vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

Does the virtualization work equally well for lots of history as it does for large working copies?

I have a 100k commit svn repo I have been trying to migrate but the result is just too large. Partly this is due to tons of revisions of binary files that must be in the repo.

Does the virtualization also help provide a shallow set of recent commits locally but keep all history at the server (which is hundreds of gigs that is rarely used)?

Re: The largest Git repo

#56

Post by mschuster91 » Wed, May 24, 2017, 5:20 PM UTC

vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

How do you prevent data exfiltration? I mean, in theory you could restrict the visibility of repos to the user based on team membership/roles and so prevent a single person from unauditably exfiltrating the whole Windows source code tree. In contrast with a monorepo there likely won't be any alerts triggered if someone does do a full git clone, except for someone saturating his switch port...

Re: The largest Git repo

#57

Post by saeednoursalehi » Wed, May 24, 2017, 5:22 PM UTC

chinhodado wrote: post #50

vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

Why do you name it "GVFS" instead of something more descriptive like "GitVFS"?

This was discussed a bit in the comments in Brian Harry's last post on GVFS: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-g...

We're building a VFS (Virtual File System) for Git (G) so GVFS was a very natural name and it just kind of stuck once we came up with it.

Re: The largest Git repo

#58

Post by MS_Buys_Upvotes » Wed, May 24, 2017, 5:23 PM UTC

Wow the majority of posts here are from Microsoft employees.

Re: The largest Git repo

#59

Post by mschuster91 » Wed, May 24, 2017, 5:23 PM UTC

kk1274 wrote: post #53

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

Hmm. I believe it's likely there's also lots of binary assets - the WAV sound files, BMP images, the "hello world" videos, for example - and possibly also the raw versions of the assets. And if it's really the whole history of Windows in there, that's a LOT of binary assets in LOTS of versions.

Re: The largest Git repo

#60

Post by taylorlafrinere » Wed, May 24, 2017, 5:23 PM UTC

kk1274 wrote: post #53

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

Most of that 300GB isn't text. There are test assets, images, videos, built binaries, vhd's, etc. Also, I should be clear that that 300GB is just at tip (no history). We can debate about whether or not those things should be checked into the repo but they are there now.