Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

21–30 of 416 posts

Re: The largest Git repo

#21

Post by [deleted] »

vtbassmatt wrote: post #15
CiPHPerCoder wrote: post #3

Earlier quoted context omitted.

Sure, a couple of questions: 1. How do you measure "largeness" of a git repo? 2. How are you confident that you have the largest? 3. How much technical debt does that translate to?

1. Saeed is writing a really nice series of articles starting here: https://www.visualstudio.com/learn/git-at-scale/ In the first one, he lays out how we think about small/medium/large repos. Summary: size at tip, size of history, file count at tip, number of refs, and number of developers. 2. Fairly confident, at least as far as usable repos go. Given how unusable the Windows repo is without GVFS and the other thing…

[deleted]

Re: The largest Git repo

#22

Post by domoritz »

Willamin wrote: post #19

I don't know much about Windows development, but I'm sure the system is modularized in some way. Why wouldn't you want to break up the project into multiple repos for different parts of the system? That would let you work on and test each part independent of the rest. Each part should be able to function on its own, right? Of course some engineers would need to build and test the entire OS as a whole, but I'd wager t…

Google has a single repo. The advantages are that you don't need to version anything because you always build against head. It's awesome but requires some discipline and good infrastructure.

Re: The largest Git repo

#23

Post by saeednoursalehi »

bluejekyll wrote: post #8
vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

I'm currently investigating using GitLFS for a large repo that has many binary and other large artifacts. I'm curious, did you experiment with LFS for prior to building GitVFS? Also, I know that there is an (somewhat) active effort to port GitVFS to Linux, do you know if any of the Git vendors (GitLab and/or GitHub) are planning to support GitVFS in their enterprise products?

Yes we did evaluate LFS. The thing about LFS is that while it does help reduce the clone size, it doesn't reduce the number of files in the repo at all. The biggest bottleneck when working with a repo of this size is that so many of your local git operations are linear on the number of files. One of the main values of GVFS is that it allows Git to only consider the files you're actually working with, not all 3M+ files in the repo.

Re: The largest Git repo

#24

Post by hyperrail »

tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

As Brian Harry alluded to, this is in fact very close to the previous system. As I recall it from my time at Microsoft:

The Windows source control system used to be organized as a large number (about 20 for product code alone, plus twice that number for test code and large test data files) of independent master source repositories.

The source checkouts from these repos would be arranged in a hierarchy on disk. For example, if root\ were the root of your working copy, files directly under root\ would come from the root repo, files under root\base\ from the base repo, root\testsrc\basetest from basetest, and so on.

To do cross repo operations you used a tool called "qx" (where "q" was the name of the basic source control client). Qx was a bunch of Perl scripts, some of which wrapped q functions and others that implemented higher-level functions such as merging branches. However, qx did not try to atomically do operations across all affected repos.

(The closest analog to this in git land would be submodules.)

While source control was organized this way, build and test ran on all of Windows as a single unit. There was investigation into trying to more thoroughly modularize Windows in the past, but I think the cost was always judged too great.

Mark Lucovsky did a talk several years ago on this source control system, among other aspects of Windows development:

https://www.usenix.org/legacy/events/usenix-win2000/tech.htm...

I believe it is still valid for folks not using GVFS to access Windows sources.

Re: The largest Git repo

#25

Post by vtbassmatt »

tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/

Summary:

- Complicates daily life for every engineer

- Becomes hard to make cross-cutting changes

- Complicates releasing the product

- There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would still have been a similar order of magnitude in most dimensions

Re: The largest Git repo

#26

Post by saeednoursalehi »

kingbirdy wrote: post #7
vtbassmatt wrote: post #2

A handful of us from the product team are around for a few hours to discuss if you're interested.

The article mentions relying on a windows filesystem driver. Two questions about that: 1) Why include it in default windows? It seems that 99.99% of users would never even know it existed, let alone use it 2) Does that mean GVFS isn't useable on *nix systems? Any plans to make it useable, if so?

1) The file system driver is called GvFlt. If it does get included in Windows by default, it'll be to make it easier for products like GVFS, but GvFlt on its own is not usable by end users directly.

2) GVFS is currently only available on Windows, but we are very interested in porting to other platforms.

Re: The largest Git repo

#27

Post by arjie »

tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Why split it into separate modules? Seeing that big companies are very successful with monorepos (Google, Facebook, Microsoft), has made me reconsider if repository modularization is actually worth it. There are a host of advantages to not modularizing repos, and I'm beginning to believe they outweigh those of modular repos.

Re: The largest Git repo

#29

Post by hyperrail »

breck wrote: post #11

This is so awesome. Brilliant move MS! In addition to enabling Windows engineers to be significantly more productive (eventually), it will go a long way to enabling engineers in other departments to contribute to Windows. For example, I used to work in the Azure org and once noticed a relatively simple missing feature in Windows. I filed a bug and was in contact with a PM who suggested if I wanted I could work on add…

I'm not so sure moving to Git alone would have helped your case. Getting an enlistment is only a small part of contributing to Windows.

Re: The largest Git repo

#30

Post by Kenji »

I am kinda surprised that Microsoft doesn't use tfs - after all, it's their own version control system. But then again, we use tfs at work and not a day goes by on which I do not long for git.
Post reply on HN