Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

241–250 of 416 posts

Re: The largest Git repo

#241

Post by cryptonector »

Jyaif wrote: post #51
howinator wrote: post #14

Earlier quoted context omitted.

So, this is actually pretty common. I know that both Google and Facebook use a huge mono-repo for literally everything (except I think Facebook split out their Android code into a separate repo?). So, all of Facebook's and Google's code for front-end, back-end, tools, infrastructure, literally everything, lives in one repo. It's news to me that Windows decided to go that route too. Personally, I think submodules and…

pros to big repo: -dont have to spend time to think about defining interfaces cons: -history is full of crap you dont care about -tests take forever to run -tooling breaks down completely, though thanks to MS the limit was increased seriously

You still have to do something about internal interfaces. The problem is that the moment you want to make a backwards-incompatible change to an internal interface now you have to go find users of it, and there go the benefits of GVFS... Or you can let the build and test system tell you what breaks (take a long coffee break, repeat as many times as it takes; could be many times). Or use something like OpenGrok to find all those uses in last night's index of the source.

Defining what portions of the OS you'll have to look in for such changes helps a great deal.

As to building and testing... the system has to get much better about detecting which tests will need to be re-run for any particular change. That's difficult, but you can get 95% of the way there easily enough.

Re: The largest Git repo

#242

Post by e40 »

cryptonector wrote: post #194

At Sun Microsystems, Inc., (RIP) we have many "gates" (repos) that made up Solaris. Cross-gate development was somewhat more involved, but still not bad. Basically: you installed the latest build of all of Solaris, then updated the bits from your clones of the gates in question. Still, a single repo is great if it can scale, and GVFS sounds great! But that's not what I came in to say. I came in to describe the rebase…

Can you describe it in a little more detail? Do you still use branches? If so, for what? For different versions?

Re: The largest Git repo

#243

Post by e40 »

vtbassmatt wrote: post #25
tobyhinloopen wrote: post #5

I wonder why Windows is a single repository - Why not split it in separate modules? I can imagine tools like Explorer, Internet Explorer/Edge, Notepad, Wordpad, Paint, etc. all can stay in its own repository. I can imagine you can even further split things up, like a kernel, a group of standard drivers, etc. If that is not already the case (separate repos, that is), are the plans to separate it in the future?

Really good question. Actually, splitting Windows up was the first approach we investigated. Full details here: https://www.visualstudio.com/learn/gvfs-design-history/ Summary: - Complicates daily life for every engineer - Becomes hard to make cross-cutting changes - Complicates releasing the product - There's a still a core of "stuff" that's not easy to tease apart, so at least one of the smaller Windows repos would…

It's harder to share code between repos, though.

EDIT: like if something was to be shared between Windows and Office, for example.

Re: The largest Git repo

#245

Post by tokenizerrr »

MikeHolman wrote: post #221
js2 wrote: post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

Generally, it's the responsibility of whoever made the changes to resolve conflicts (basically, git blame conflicting lines and the people who changed them get notified to resolve conflicts in that file). Distributing the work like this makes the merges more reasonable.

How does that work?

Re: The largest Git repo

#246

Post by babuskov »

lilyball wrote: post #181
js2 wrote: post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

For 3-way merging the best tool I've found is Steve Losh's splice.vim ( https://github.com/sjl/splice.vim/ ).

My favorite tool is Meld:

http://meldmerge.org/

It's open source. Also:

https://stackoverflow.com/questions/572237/whats-the-best-th...

Re: The largest Git repo

#247

Post by flai »

Jach wrote: post #203
Eyas wrote: post #180

Earlier quoted context omitted.

You say that, but it is very telling that every large company out there (Google and Facebook come to mind) go for the single-repository approach. I'm sure that, when dealing with stakeholder structures where different organizations can depend on different bits and pieces, having multiple repositories with difficulty of making breaking and cross-cutting changes, becomes good. From the view of a single organization whe…

It is very telling. Google has a cloud supercomputer doing nothing but building code to support their devs. I don't know about Facebook. (I really don't -- I'm constantly amazed that they have as many engineers that they do, what do they all work on?) Where I work ( https://medium.com/salesforce-engineering/monolith-to-micros... ) there's a big monolith but with more push towards breaking things up, at the architectu…

You forget Conway's law:

> organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations

Open source has a very different communication structure than a company. While the big three (MS, Google, FB) try to work towards good inter-departmemt relations, it is usually either

- a single person - a small group

that are the gatekeepers for a small amout of code, typically encapsulated in a "project". They do commit a lot to their project, yet rarely touch other projects in comparision.

Also, collaboration is infinitely harder, as in the office you can simply walk up to someome, call them, or chat them - in OSS a lot of communication works via Issues and PRs, which are a fundamentally different way to communicate.

This all is reflected by the structure of how a functionality is managed: Each set of gatekeepers gets their own repository, for their function.

Interestingly this even happens with bigger repositories: DefinitelyTyped is a repository for TypeScript "typings" for different JS libraries, which has hundreds of collaborators.

Yet, if you open a pull-request for an existing folder the ones that have previously made big-ish changes can approve / decline the PR, so each folder is its own little repo.

https://github.com/DefinitelyTyped/DefinitelyTyped/pull/1672...

So: maybe the solution is big repos for closed companies, small repos for open-source?

Re: The largest Git repo

#248

Post by cryptonector »

comex wrote: post #173
lloeki wrote: post #125

Coming from the days of CVS and SVN, git was a freaking miracle in terms of performance, so I have to just put things into perspective here when the topmost issue of git is performance . It's just a testament how huge are the codebases we're dealing with (Windows over there, but also Android, and surely countless others), the staggering amount of code we're wrangling around these days and the level of collaboration i…

Broadly speaking this is true, but note that in some ways CVS and SVN are better at scaling than Git. - They support checking out a subdirectory without downloading the rest of the repo, as well as omitting directories in a checkout. Indeed, in SVN, branches are just subdirectories, so almost all checkouts are of subdirectories. You can't really do this in Git; you can do sparse checkouts (i.e. omitting things when c…

Nothing about git prevents an "ask the remote" feature. It's just not there. I suspect that as git repos grow huge and shallow and partial cloning becomes more common, git will grow such a feature. Granted, it doesn't have it today. And the GVFS thing is... a bit of a hack around git not having that feature -- but it proves the point.

Re: The largest Git repo

#249

Post by dom0 »

cryptonector wrote: post #208
holtalanm wrote: post #201

Earlier quoted context omitted.

we use a rebase workflow in git at my current employer, and it is amazing. previous employer used a merge workflow (primarily because we didnt understand git very well at the time), and there were merge conflicts all the time when pulling new changes down or merging new changes in. It was a headache to say the least. As the integration manager for one project, I usually spent the better part of an hour just going thr…

Yes! One of the most important advantages of a rebase workflow is that you can see immediately what upstream commits your conflict with, as opposed to some massive merge you have to go chasing branch history to figure out the semantics of the change in question. "Amazing" is right. Sun was doing rebases in the 90s, and it never looked back.

My exact experience (in the context of "merging upwards"). Large merges are a huge pain to do, and are basically impossible to review, too.

Re: The largest Git repo

#250

Post by e40 »

vtbassmatt wrote: post #93
asdfgadsfgasfdg wrote: post #80

Earlier quoted context omitted.

What would someone do with the source code for Windows? No one in open source would want to touch it. No large company would want to touch it. Grey/black hats are probably happier with their decompilers. Surely it would be easier to pirate than build (assuming their build system scales with most build systems I've observed in the wild). No small company would want to touch it. Anyway MS share source with various thir…

This is all correct. Also, we'd notice someone grabbing the whole 300GB whether it's in 40 SD depots or a single Git repo.

Mind if I ask how you'd notice?
Post reply on HN