Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

121–130 of 416 posts

Re: The largest Git repo

#121

Post by RUTHLESS_RUFUS »

cosinetau wrote: post #99
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

This was a very interesting point. It sounds like there are some serious architectural limitations on Windows, and this makes me believe the same might be true for the NT kernel, and that MS might not be interested in doing heavy refactoring of it. I'm not a frequent Windows user, or a Windows dev at all. Does anyone know of any consequences that MS's decision might mean, if this hypothesis is true?

hey chummer, show me your elegant repos will ya?

Re: The largest Git repo

#122

Post by vtbassmatt »

alkonaut wrote: post #111
taylorlafrinere wrote: post #60

Earlier quoted context omitted.

Most of that 300GB isn't text. There are test assets, images, videos, built binaries, vhd's, etc. Also, I should be clear that that 300GB is just at tip (no history). We can debate about whether or not those things should be checked into the repo but they are there now.

How did you go about creating the central repo and how long did it take? A 2Gb at tip svn repo with 100k commits is taking me many days and each odd failure typically has me restart the process after filtering out some obscure part of the tree. Edit: read in another comment that you dropped the history. Understandable, but can appreciate how that would add to friction (devs having to look through two different histor…

The Windows team developed a tool called "GitTrain" that knew how to:

- migrate the tip of a branch to Git (yes, the 300GB number is the tips of all the interesting branches, not the history)

- keep a Git branch and a SD branch in sync for a while

- be re-run over each of the 400+ branches they care about

But they went through some of the same trial-and-error process that you're describing.

Re: The largest Git repo

#123

Post by jonathanpoulter »

asp2insp wrote: post #110
jonathanpoulter wrote: post #94

Earlier quoted context omitted.

How does a mono-repo company manage open sourcing a single part of their infrastructure if things are in one large repo? For example, if everything lived in one repo, how does Facebook manage open sourcing React? Or if I personally wanted to switch to one private mono-repo, how would I share individual projects easily?

We have an internal tool that allows us to mirror subdirectories of our monorepo into individual github repositories, and another tool that helps us sync our internal source code review tool with PRs etc.

An internal tool which manages commits, between individual repos etc. does it not seem that this is a logical extension to git itself? A little like submodules, but being able to publish only parts of the sourcetree. Maybe it would be impossible to keep any consistency and leaking information from the rest of the tree.

Re: The largest Git repo

#124

Post by je42 »

arjie wrote: post #27
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Why split it into separate modules? Seeing that big companies are very successful with monorepos (Google, Facebook, Microsoft), has made me reconsider if repository modularization is actually worth it. There are a host of advantages to not modularizing repos, and I'm beginning to believe they outweigh those of modular repos.

mono repo only works if you have tooling. google's and facebook's tools are not opensource. also, ms tooling is windows only.

so for most of us the only reasonable path is to split into multiple repositories.

it is also easier to create tools that deal with many repositories.

than it is to create a tool that virtualizes a single large repo.

Re: The largest Git repo

#125

Post by lloeki »

Coming from the days of CVS and SVN, git was a freaking miracle in terms of performance, so I have to just put things into perspective here when the topmost issue of git is performance. It's just a testament how huge are the codebases we're dealing with (Windows over there, but also Android, and surely countless others), the staggering amount of code we're wrangling around these days and the level of collaboration is incredible and I'm quite sure we would not have been able to do that (or at least not that nimbly and with such confidence) were it not for tools like git (and hg). There's a sense of scale regarding that growth across multiple dimensions that just puts me in awe.

Re: The largest Git repo

#126

Post by erikpukinskis »

vtbassmatt wrote: post #25
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes

This does seem like a negative, doesn't it?

But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo.

It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be.

It puts pressure on you to really, actually separate concerns. Not just put "concerns" into a source file that can reach into any of a million other source files and twiddle the bits.

"Easy to make sweeping changes" really means "easy to limp along with a bad architecture."

I think that's one of the reasons why so much code rots: developers thinking it should be easy to make arbitrary changes.

No, it should be hard to make arbitrary changes. It should be easy to make changes with very few side effects, and hard to make changes that affect lots of other code. That's how you get modules that get smaller and smaller, and change less and less often, while still executing often. That's the opposite of code rot: code nirvana.

Re: The largest Git repo

#127

Post by sriram_sun »

garyclarke27 wrote: post #45

Linus must be very proud - his favourite software Windows - now depends on GIT.

Well how the tables have turned! Only about 3 yrs back I was having a conversation with a Microsoft engineer about them evaluating a closed source Hadoop clone because Microsoft policy prohibited them from using open source.

Re: The largest Git repo

#128

Post by wilatmsft »

ATsch wrote: post #91
saeednoursalehi wrote: post #26

Earlier quoted context omitted.

1) The file system driver is called GvFlt. If it does get included in Windows by default, it'll be to make it easier for products like GVFS, but GvFlt on its own is not usable by end users directly. 2) GVFS is currently only available on Windows, but we are very interested in porting to other platforms.

What are your thoughts on implementing something more general like linux's FUSE[1] instead? A general Virtual Filesystem driver in Windows could be used for a wide range of things and means you don't just have a single-purpose driver sitting around. [1] https://en.m.wikipedia.org/wiki/Filesystem_in_Userspace

A general FUSE-like API would be very useful to have, but unfortunately it can't meet our performance requirements.

The first internal version of GVFS was actually based on a 3rd party driver that looks a lot like FUSE. But because it is general purpose, and requires all IO to context switch from the kernel to user mode, we just couldn't make it fast enough.

Remember that our file system isn't going to be used just for git operations, but also to run builds. Once a file has been downloaded, we need to make sure that it can be read as fast as any local file would be, or dev productivity would tank.

With GvFlt, we're able to virtualize directory enumeration and first-time file reads, but after that get completely out of the way because the file becomes a normal NTFS file from that point on.

Re: The largest Git repo

#129

Post by maxxxxx »

smilekzs wrote: post #103
maxxxxx wrote: post #73

Earlier quoted context omitted.

Here is my cynical view: From what I know they have a history of not using their own tools. They didn't use SourceSafe, but Perforce. Then they made an effort to switch to TFS, realized that it sucks and moved on to git. You can see the same pattern in Windows desktop apps. They didn't use MFC for themselves, didn't use Winforms, used WPF only a little.

UWP everywhere now though.

Is that true? Which larger app is written in UWP? Something like Office, Skype or Visual Studio.

Re: The largest Git repo

#130

Post by amptorn »

arjie wrote: post #27
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Why split it into separate modules? Seeing that big companies are very successful with monorepos (Google, Facebook, Microsoft), has made me reconsider if repository modularization is actually worth it. There are a host of advantages to not modularizing repos, and I'm beginning to believe they outweigh those of modular repos.

> Why split it into separate modules?

Well, because of the unbelievable amount of engineering work involved in trying to get Git to operate at such insane scale? To say nothing of the risk involved in the alternative. This project in particular could easily have been a catastrophe.

Post reply on HN