Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The largest Git repo

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

261–270 of 416 posts

Re: The largest Git repo

#261

Post by sriram_sun »

tonmoy wrote: post #135
sriram_sun wrote: post #127

Earlier quoted context omitted.

Well how the tables have turned! Only about 3 yrs back I was having a conversation with a Microsoft engineer about them evaluating a closed source Hadoop clone because Microsoft policy prohibited them from using open source.

Using open source in their product is different from using it for development. I wonder which one you talking about with that employee

I'm pretty sure it was for a product (server side), not an internal dev tool. Back in 2000 one of my roommates was an intern with the VS team and he said a lot of the devs were using emacs.

Re: The largest Git repo

#262

Post by MaulingMonkey »

zanny wrote: post #69
kk1274 wrote: post #53

300GB of code WOW! Just for comparison the entire English Wikipedia dump including all media is about 50-60GB. What are you guys doing there and how large do you see this growing?

A lot of that is probably static art assets. A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files. It is part of the reason why the Chrome or Unreal Engine repos are preposterously large (hundreds of megs to gigabytes each).

> A lot of big corporate projects are guilty of just throwing everything in one tree like this where for some bizarre reason you are checking out jpgs and video files.

Would you want to build chrome without it's icons for bookmark folders, the launcher, etc.? No?

What about a game without it's intro videos? Even if you need to debug a crash during the initial intro, related to playing said video?

Some people prefer this kind of content to be managed with a different, media focused version control system, or a separate tree, but now you have 2+ disparate distribution / version control systems to try and keep in sync. You do have options like git submodules now for some VCSes, but I haven't been impressed with their workflow, and they weren't always an option. I will gladly burn several gigs just to simplify my workflow - I've got the spare SSD and bandwidth. I've had fully built checkouts in the hundreds of gigabytes range.

And don't get me wrong, "hundreds of gigabytes" is getting annoyingly large if the IT department skimped and bought me a 250GB SSD. I'll abuse directory junctions to offload some of that onto mechanical drives. This is relatively fire and forget though - multi-repository is not.

Re: The largest Git repo

#263

Post by pmontra »

hinkley wrote: post #210
daxfohl wrote: post #197

Earlier quoted context omitted.

I'd rather see google open up their monorepo as a platform, and compete with github. git is fine, but there's something compelling about a monorepo. Whether they do it one-monorepo-per-account, or one-global-monorepo, or some mix of the two, would be interesting to see how it shapes up.

This is the main thing I miss about subversion. You could check out any arbitrary subdirectory of a repository. On two projects the leads and full stack people had the whole thing checked out, everybody else just had the one submodule they were responsible for. Worked fairly well.

Git has submodules too and teams usually have access control on the main server used for sharing commits.

Re: The largest Git repo

#264

Post by sriram_sun »

vtbassmatt wrote: post #156
sriram_sun wrote: post #127

Earlier quoted context omitted.

Well how the tables have turned! Only about 3 yrs back I was having a conversation with a Microsoft engineer about them evaluating a closed source Hadoop clone because Microsoft policy prohibited them from using open source.

Different divisions have had different stances on open source code for a long time. Somewhere I still have the t-shirt from our first "Open Source Day" event back in 2008 (and it's not like that was the first time any MS employee had ever considered using open source). Things are a lot more standardized now, with a big push from both the top and the bottom to use open source wherever it makes sense. Why reinvent the…

> Why reinvent the wheel?

Look who's talking :)!

Wasn't WiX released and developed as an Open Source project by MS?

Re: The largest Git repo

#265

Post by cryptonector »

e40 wrote: post #242
cryptonector wrote: post #194

At Sun Microsystems, Inc., (RIP) we have many "gates" (repos) that made up Solaris. Cross-gate development was somewhat more involved, but still not bad. Basically: you installed the latest build of all of Solaris, then updated the bits from your clones of the gates in question. Still, a single repo is great if it can scale, and GVFS sounds great! But that's not what I came in to say. I came in to describe the rebase…

Can you describe it in a little more detail? Do you still use branches? If so, for what? For different versions?

Great question.

We basically had a single branch per repo, and every repo other than the master one was a fork (ala github). But that was dictated by the limitations of the VCS we used (Teamware) before the advent of Hg and git.

So "branches" were just a developer's or project's private playgrounds. When done you pushed to the master (or abandoned the "branch"). Project branches got archived though.

In a git world what this means is that you can have all the branches you want, and you can even push them to the master repo if that's ok with its maintainers, or else keep them in your forks (ala github) or in a repo meant for archival.

But! There is only one true repo/branch, and that's the master branch in the master repo, and there are no merge commits in there.

For developers working on large projects the workflow went like this:

- clone the project repo/branch

- work and commit, pulling --rebase periodically

- push to the project repo/branch

- when the project repo/branch rebases onto a newer upstream the developer has to rebase their downstream onto the rebased project repo/branch

Project techleads or gatekeepers (larger projects could have someone be gatekeeper but not techlead) would be responsible for rebasing the project onto the latest upstream.

To simplify things the upstream did a bi-weekly "release" (for internal purposes) that projects would rebase onto on either a bi-weekly or monthly schedule. This minimizes the number of rebases to do periodically.

When the project nears the dev complete time, the project will start rebasing more frequently.

For very large projects the upstream repo would close to all other developers so that the project could rebase, build, test, and push without having to rinse and repeat.

(Elsewhere I've seen uni-repo systems where there is no closing of the upstream for large projects. There a push might have to restart many times because of other pushes finishing before it. This is a terrible problem. But manually having to "close" a repo is a pain too. I think that one could automate the process of prioritizing pushes so as to minimize the restarts.)

Re: The largest Git repo

#266

Post by erikpukinskis »

Goronmon wrote: post #153
erikpukinskis wrote: post #126

Earlier quoted context omitted.

> - Becomes hard to make cross-cutting changes This does seem like a negative, doesn't it? But it's not. Making it hard to make cross-cutting changes is exactly the point of splitting up a repo. It forces you to slow down, and—knowing that you can only rarely make cross-cutting changes—you have a strong incentive to move module boundaries to where they should be. It puts pressure on you to really, actually separate c…

No, it should be hard to make arbitrary changes. If you change the word "arbitrary" to "necessary" (implying a different bias than the one you went with) then all of a sudden this attitude sounds less helpful. Similarly "easy to limp along with a bad architecture" could be re-written as "easy to work with the existing architecture". At the end of the day, it's about getting work done, not making decisions that are th…

> At the end of the day, it's about getting work done, not making decisions that are the most "pure".

This attitude will lead to a total breakdown of the development process over the long term. You are privileging Work Done At The End Of The Day over everything else.

You need to consider work done at every relevant time scale.

How much can you get done today?

How much can you get done this month?

How much can you get done in 5 years?

Ignore any of these questions at your peril. I fundamentally agree with you about purity though. I'm not sure what in my piece made you think I think Purity Uber Alles is the right way to go.

Re: The largest Git repo

#267

Post by bitJericho »

bronson wrote: post #240
bitJericho wrote: post #138

Earlier quoted context omitted.

This just blew my mind. I'm gonna go home and see about combining all my projects. That seems very useful!

You don't have the same requirements as Google.

What are my requirements such that this wouldn't work for me?

Re: The largest Git repo

#268

Post by cromwellian »

monksy wrote: post #131
planteen wrote: post #104

Earlier quoted context omitted.

Google also uses a single giant repo...

Why is that a "good reason" to do it?

It radically simplifies everything. Every commit is reproducible across the entire tool chain and ecosystem.

It makes the entire system kind of pure-functional and stateless/predictable. Everything from computing which tests you need to run, to who to blame when something breaks, to caching build artifacts, or even sharing workspaces with co-workers.

While this could be implemented with multiple repros underneath, it would add much complexity.

Re: The largest Git repo

#269

Post by draw_down »

js2 wrote: post #145

Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts). At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours. Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my t…

God, that sounds hellish.

Re: The largest Git repo

#270

Post by cryptonector »

nikital wrote: post #254
cryptonector wrote: post #194

At Sun Microsystems, Inc., (RIP) we have many "gates" (repos) that made up Solaris. Cross-gate development was somewhat more involved, but still not bad. Basically: you installed the latest build of all of Solaris, then updated the bits from your clones of the gates in question. Still, a single repo is great if it can scale, and GVFS sounds great! But that's not what I came in to say. I came in to describe the rebase…

A pain I have with rebase workflow is that it creates untested commits (because diffs were blindly applied to a new version of the code). If I rebase 100 commits, some of the commits will be subtly broken. How do you deal with that?

With git rebase you can in fact build and test each commit. That's what the 'exec' directive is for (among other things) in rebase scripts!

Basically, if you pick a commit, and in the next line exec make && make check (or whatever) then that build & test command will run with the workspace HEAD at that commit. Add such an exec after every pick/squash/fixup and you'll build and test every commit.

Post reply on HN