Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

171–180 of 416 posts

Re: The largest Git repo

#171
post #167
post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

When you have that many conflicts, it's often due to massive renames, or just code moves. If you use git-mediate[1], you can re-apply those massive changes on the conflicted state, run git-mediate - and the conflicts get resolved. For example: if you have 300 conflicts due to some massive rename, you can type in: git-search-replace.py[2] -f oldGlobalName///newGlobalName git-mediate -d Succcessfully resolved 377 confl…

[deleted]

Re: The largest Git repo

#172
post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

Windows developed an extension that lets them do conflict resolution in the web. We have a server-side API that it calls into, but the extension isn't fundamentally different from using BeyondCompare or $YOUR_FAVORTE_MERGETOOL.

Re: The largest Git repo

#173
post #125

Coming from the days of CVS and SVN, git was a freaking miracle in terms of performance, so I have to just put things into perspective here when the topmost issue of git is performance . It's just a testament how huge are the codebases we're dealing with (Windows over there, but also Android, and surely countless others), the staggering amount of code we're wrangling around these days and the level of collaboration i…

Broadly speaking this is true, but note that in some ways CVS and SVN are better at scaling than Git.

- They support checking out a subdirectory without downloading the rest of the repo, as well as omitting directories in a checkout. Indeed, in SVN, branches are just subdirectories, so almost all checkouts are of subdirectories. You can't really do this in Git; you can do sparse checkouts (i.e. omitting things when copying a working tree out of .git), but .git itself has to contain the entire repo, making them mostly useless.

- They don't require downloading the entire history of a repo, so the download size doesn't increase over time. Indeed, they don't support downloading history: svn log and co. are always requests to the server. Unfortunately, Git is the opposite, and only supports accessing previously downloaded history, with no option to offload to a server. Git does have the option to make shallow clones with a limited amount of (or no) history, and unlike sparse checkouts, shallow clones truly avoid downloading the stuff you don't want. But if you have a shallow clone, git log, git blame, etc. just stop at the earliest commit you have history for, making it hard to perform common development tasks.

I don't miss SVN, but there's a reason big companies still use gnarly old systems like Perforce, and not just because legacy: they're genuinely much better at scaling to huge repos (as well as large files). Maybe GVFS fixes this; I haven't looked at its architecture. But as a separate codebase bolted on to near-stock Git, I bet it's a hack; in particular, I bet it doesn't work well if you're offline. I suspect the notion of "maybe present locally, maybe on a server" needs to be baked into the data model and all the tools, rather than using a virtual file system to just pretend remote data is local.

Re: The largest Git repo

#174
Any word on open sourcing parts of the windows OS now that MS is seeing the light? The head guys have to see the benefits by now.

It says something that MS chose Git over anything proprietary that they developed.

Re: The largest Git repo

#175

A handful of us from the product team are around for a few hours to discuss if you're interested.

Sure, a couple of questions: 1. How do you measure "largeness" of a git repo? 2. How are you confident that you have the largest? 3. How much technical debt does that translate to?

I recently heard a story that someone tried to push a 1TB repo to our university Gitlab which then ran out of disk space. Sure, that might have been not be a usable repo but only an experiment. Still, I would bet against the claim that 300GB is the largest one.

Re: The largest Git repo

#176
post #167
post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

When you have that many conflicts, it's often due to massive renames, or just code moves. If you use git-mediate[1], you can re-apply those massive changes on the conflicted state, run git-mediate - and the conflicts get resolved. For example: if you have 300 conflicts due to some massive rename, you can type in: git-search-replace.py[2] -f oldGlobalName///newGlobalName git-mediate -d Succcessfully resolved 377 confl…

I just looked at git-mediate and I'm very confused. It appears that all it does is remove the conflict markers from the file after you've already manually fixed the conflict. Except you need to do more work than normal, because you need to apply one of the changes not only to the other branch's version but also to the base. What am I missing here, why would I actually want to use git-mediate when I'm already doing all the work of resolving the conflicts anyway?

Re: The largest Git repo

#177
post #105

Earlier quoted context omitted.

-dont have to spend time to think about defining interfaces That seems like a design and policy choice, orthogonal to repos.

Not really. It's easier to make a single atomic breaking change to how different components talk to each other if they are in the same repository. If they are in different repos, the change is not atomic and you need to version interfaces or keep backwards compatibility in some other way.

It's very much really. The fact that it's easier doesn't really matter - a repo is about access to the source code and its history with some degree of convenience. The process and policy of how you control actual change is quite orthogonal. You can have a single repo and enforce inter-module interfaces very strongly. You can have 20 repos and not enforce them at all. Same goes for builds, tests, history, etc. The underlying technology can influence the process but it doesn't make it.

Re: The largest Git repo

#178
post #166
post #72

Earlier quoted context omitted.

It makes total sense when the expectation is that any engineer in the company can build any part of the stack at any time with a minimum of drama.

What's dramatic about copy and pasting a clone uri into a command?

A url? Not much.

100 urls? That's getting a bit annoying.

Re: The largest Git repo

#179

Any word on open sourcing parts of the windows OS now that MS is seeing the light? The head guys have to see the benefits by now. It says something that MS chose Git over anything proprietary that they developed.

I'm guessing that would be a licensing nightmare. They must pay many companies for licensed technologies inside Windows, and many of those licenses likely wouldn't be compatible with open source licensing.

All of their source code would have to go through legal review, some with each check in. I don't see that happening for legacy code.

Re: The largest Git repo

#180

Earlier quoted context omitted.

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

You say that, but it is very telling that every large company out there (Google and Facebook come to mind) go for the single-repository approach.

I'm sure that, when dealing with stakeholder structures where different organizations can depend on different bits and pieces, having multiple repositories with difficulty of making breaking and cross-cutting changes, becomes good.

From the view of a single organization where the only users of a component are other components in the same organization, it seems like there is consensus around single-repository.

Post reply on HN