Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

141–150 of 416 posts

Re: The largest Git repo

#141
post #131

Earlier quoted context omitted.

Google also uses a single giant repo...

Why is that a "good reason" to do it?

Not really :) and I am not the OP. This article [1] provides a very good overview of the repository organization Google has and the reasons behind it.

I think the reason that this works for well Google is the amount of test automation that is in place which seems to be very good at flagging any errors due to dependency changes before it gets deployed. Not sure how many organizations have invested and have an automated test infrastructure like Google has built.

1. https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

Re: The largest Git repo

#142
Those of us working on smaller codebases may wonder what the big deal is. Facebook had a similar problem leading them to switch out to mercurial. https://code.facebook.com/posts/218678814984400/scaling-merc... It is awesome that the problems could be solved in git itself.

Also, kudos to the writer of the blog. It is a really high quality blog post. The percentile measures of performance, survey responses from users etc are very typical of solid incremental approaches to challenges faced by startups except these are internal customers.

Re: The largest Git repo

#143

What was performance like for the Source Depot system? It would be interesting to note the comparison between the old SDX system and GVFS.

Quoting Brian Harry from a comment response at https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-large...

"It depends a great deal on the operation. SourceDepot was much faster at some things – like “sd opened”, the equivalent of “git status”. sd opened was On an operation, by operation basis, SD is still much faster than our Git/GVFS solution. We're still working on it to close the gap but I'm not sure it will ever get as fast at everything. The broader question, though is about overall developer productivity and we think we are on a path to winning that."

Re: The largest Git repo

#144

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

I have worked at a company using one repo per team and at Google which uses a big monorepo. I much prefer the latter. As long as you have the infrastructure to support it, I see no other downsides (obviously, Google does not use git).

Re: The largest Git repo

#145
Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts).

At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours.

Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my time. Ensuring the code compiled was another 5 minutes to an hour. If someone on the Chromium team had significantly refactored a component, which typically occurred every couple weeks, I knew half my day was going to be spent dealing with the refactor.

The Chromium team at the time was many dozens of engineers, landing on the order of a hundred commits per day. Our team was a dozen engineers landing maybe a couple dozen commits daily. A large merge might have on the order of 100 conflicts, but typically it was just a dozen or so conflicts.

Which is to say: I don't understand how it's possible to deal with a merge that has 1k conflicts across 10k changes. How often does this occur? How many people are responsible for handling the merge? Do you have a way to distribute the conflict resolution across multiple engineers, and if so, how? And why don't you aim for more frequent merges so that the conflicts aren't so large?

(And also, your merge tool must be incredible. I assume it displays a three-way diff and provides an easy way to look at the history of both the left and right sides from the merge base up to the merge, along with showing which engineer(s) performed the change(s) on both sides. I found this essential many times for dealing with conflicts, and used a mix of the git CLI and Xcode's opendiff, which was one of the few at the time that would display a proper three-way diff.)

Re: The largest Git repo

#146
post #84

If you have a large repo, it actually helps you to save storage if you don't need the full history on clone by using shallow clone. hg is still behind this, AFAI can tell from search. FB has this as an extension. https://bitbucket.org/facebook/hg-experimental/ Is FB fully using hg internally, or both Git and hg, because obviously FB has public repo on Github.

My understanding is that fb is actively promoting hg for internal repositories. Not sure how they sync between public git and internal hg.

Re: The largest Git repo

#147

Earlier quoted context omitted.

Whoa. 300GB with a shallow clone?! What size does the whole repo use on the server side?

The pack file size for a full clone is 187GB. The 300GB is the working directory. We did not import the history of the code base, so the current repo only has about 5 months of history. As others have called out, there are a lot of assets in the repo that don't compress.

Why only 5 months? Will more of the history be added to the git repository eventually?

Re: The largest Git repo

#149
post #125

Coming from the days of CVS and SVN, git was a freaking miracle in terms of performance, so I have to just put things into perspective here when the topmost issue of git is performance . It's just a testament how huge are the codebases we're dealing with (Windows over there, but also Android, and surely countless others), the staggering amount of code we're wrangling around these days and the level of collaboration i…

I agree. I really think Linux needs a Nobel Price

Re: The largest Git repo

#150

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

As the linked article says, modularizing and living in separate repos was the plan of record for a while. But after evaluating the tradeoffs, we decided that Windows needs to optimize for big, rapid refactors at this stage in its development. "Easy to make sweeping changes" also means "easy to clean up architecture and refactor for cleaner boundaries".

The Windows build system is where component boundaries get enforced. Having version control introduce additional arbitrary boundaries makes the problem of good modularity harder to solve.

Post reply on HN