Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

151–160 of 416 posts

Re: The largest Git repo

#151

Post by phkahler »

erikpukinskis wrote: post #126
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

Remember when they said IE was an integral part of the OS? Yeah...

Re: The largest Git repo

#152

Post by riffraff »

je42 wrote: post #124
arjie wrote: post #27

Earlier quoted context omitted.

Why split it into separate modules? Seeing that big companies are very successful with monorepos (Google, Facebook, Microsoft), has made me reconsider if repository modularization is actually worth it. There are a host of advantages to not modularizing repos, and I'm beginning to believe they outweigh those of modular repos.

mono repo only works if you have tooling. google's and facebook's tools are not opensource. also, ms tooling is windows only. so for most of us the only reasonable path is to split into multiple repositories. it is also easier to create tools that deal with many repositories. than it is to create a tool that virtualizes a single large repo.

but google's and facebook's repos are also orders of magnitude larger than what most people deal with, normal tools might work just fine in most cases.

Re: The largest Git repo

#153

Post by Goronmon »

erikpukinskis wrote: post #126
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

No, it should be hard to make arbitrary changes.

If you change the word "arbitrary" to "necessary" (implying a different bias than the one you went with) then all of a sudden this attitude sounds less helpful.

Similarly "easy to limp along with a bad architecture" could be re-written as "easy to work with the existing architecture".

At the end of the day, it's about getting work done, not making decisions that are the most "pure".

Re: The largest Git repo

#154

Post by vtbassmatt »

bokchoi wrote: post #147
wilatmsft wrote: post #101

Earlier quoted context omitted.

The pack file size for a full clone is 187GB. The 300GB is the working directory. We did not import the history of the code base, so the current repo only has about 5 months of history. As others have called out, there are a lot of assets in the repo that don't compress.

Why only 5 months? Will more of the history be added to the git repository eventually?

No, we'll keep the SD servers around for a while for servicing older products. We also have a "breadcrumbing" system that lets an engineer follow a file's history back from Git to the old system.

Re: The largest Git repo

#155

Post by malnourish »

js2 wrote: post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

For 3 way merging, I've had good luck with beyondcompare

Re: The largest Git repo

#156

Post by vtbassmatt »

sriram_sun wrote: post #127
garyclarke27 wrote: post #45

Linus must be very proud - his favourite software Windows - now depends on GIT.

Well how the tables have turned! Only about 3 yrs back I was having a conversation with a Microsoft engineer about them evaluating a closed source Hadoop clone because Microsoft policy prohibited them from using open source.

Different divisions have had different stances on open source code for a long time. Somewhere I still have the t-shirt from our first "Open Source Day" event back in 2008 (and it's not like that was the first time any MS employee had ever considered using open source). Things are a lot more standardized now, with a big push from both the top and the bottom to use open source wherever it makes sense. Why reinvent the wheel?

Re: The largest Git repo

#158

Post by likpok »

erikpukinskis wrote: post #126
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

From my experience with Java-style hard module dependencies, this makes it extremely difficult to refactor anything touching external interfaces.

You say this forces you to think ahead, but predicting the future is quite difficult. The result is that you limp along with known-broken code because it would take so much effort to make the breaking changes to clean it up.

For example, lets say you discover that people are frequently misuing a blocking function because they don't realize that it blocks.

Let's say that we have a function `bool doThing()`. We discover that the bool return type is underspecified: there's a number of not-exactly-failure not-exactly-success cases. In a monorepo, it's pretty easy to modify this so that `doThing()` can return a `Result` instead. With multiple repos and artifacts, you either bring up the transitive closure of projects, or you leave it for someone to do later. For a widely used function, this can be prohibitive. That makes people frequently choose the "rename and deprecate" model, which means you get an increasing pile of known-bad functions.

Re: The largest Git repo

#159

Post by nxc18 »

cosinetau wrote: post #99
vtbassmatt wrote: post #25

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

This was a very interesting point. It sounds like there are some serious architectural limitations on Windows, and this makes me believe the same might be true for the NT kernel, and that MS might not be interested in doing heavy refactoring of it. I'm not a frequent Windows user, or a Windows dev at all. Does anyone know of any consequences that MS's decision might mean, if this hypothesis is true?

The NT kernel is surprisingly small and well-factored to begin with - it is a lot closer to a 'pure' philosophy (e.g. Microkernel) than something like Linux to begin with.

If you have a problem with Windows being overcomplicated or in need of refactor it is almost certainly something to do with not-the-kernel.

If you look at something like the Linux kernel its actually much larger than Windows. It needs to have every device driver known to man (except that one WiFi/GPU/Ethernet/Bluetooth driver you need) because internally the architecture is not cleanly defined and kernel changes also involve fixing all the broken drivers.

Re: The largest Git repo

#160

Post by vtbassmatt »

svanwaa wrote: post #109

Can you go into any more detail of the breakdown of your repo structure? Thanks!

edit: forgot, no Markdown here

Do you mean across all of Microsoft? Different teams have different structures. Speaking only for TFS and VSTS, we have a single repo containing the code for both, a handful of "adjunct" repos containing tools like GVFS, a repo for the documentation [1], and a bunch of open source repos for the build and release agent [2], agent tasks [3], API samples [4], and probably more I don't know about.

[1] https://www.visualstudio.com/docs

[2] https://github.com/microsoft/vsts-agent

[3] https://github.com/Microsoft/vsts-tasks

[4] https://github.com/Microsoft/vsts-dotnet-samples

Post reply on HN