Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

131–140 of 245 posts

Re: Scaling Mercurial at Facebook

#131

Post by rbehrends »

JoshTriplett wrote: post #121
mixmastamyk wrote: post #31

Earlier quoted context omitted.

I might have read this wrong, but safety for newbies doesn't sound all that bad to me.

History editing in git is not unsafe: because it's a part of the default set of tools, git offers functionality like the reflog to keep track of old history locally and avoid losing it. It's extremely difficult to permanently lose content in a git repository even using the history editing commands; it's always just a "git reflog" away.

History editing in Git is actually perfectly safe. The old commits will (at least initially) still be there, though you may need "git fsck --unreachable" to locate them if you don't remember the hashes or have them in your reflog anymore.

What is unsafe in Git is garbage collection. Commits that aren't reachable from either a branch or the reflog will be deleted after a grace period (assuming that you don't disable garbage collection entirely). More importantly, unreachable commits will also not be pushed with "git push" (which means that if your laptop blows up, you may not be able to recover them from a repository server).

This happens not directly as a consequence of history editing, but generally because a branch is being pointed to a new commit or removed (which also implicitly happens during history editing).

Re: Scaling Mercurial at Facebook

#133

Post by khuey »

dubcanada wrote: post #120
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine. But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

You really believe that Facebook couldn't find a competent C programmer to make some changes to git?

Re: Scaling Mercurial at Facebook

#134

Post by jordigh »

JoshTriplett wrote: post #121
mixmastamyk wrote: post #31

Earlier quoted context omitted.

I might have read this wrong, but safety for newbies doesn't sound all that bad to me.

History editing in git is not unsafe: because it's a part of the default set of tools, git offers functionality like the reflog to keep track of old history locally and avoid losing it. It's extremely difficult to permanently lose content in a git repository even using the history editing commands; it's always just a "git reflog" away.

It's not technically unsafe, but it's unsafe from a UI point of view. It obviously confuses some inexperienced people (witness the confusion in this thread), making them feel helpless and like their work has been destroyed, even if it hasn't. Both git and hg actually have lots of fallbacks so that work is never lost, but without experience, these fallbacks may not be obvious.

Hg's rewriting extensions are sometimes disabled by default for this reason: until people understand how they work, they shouldn't use them. If they try to use them, hg tells them how to enable these features.

Re: Scaling Mercurial at Facebook

#135

Post by jaseemabid »

dubcanada wrote: post #120
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine. But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

Its still one of the coolest and best paid companies out there! You really think they couldn't hire a few guys from core git team saying - "Dude, just improve git and we'll pay you?". You must be kidding

Re: Scaling Mercurial at Facebook

#136

Post by zaphar »

igravious wrote: post #94
zaphar wrote: post #71

Earlier quoted context omitted.

We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywh…

What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

This question is actually really complicated to answer. The short answer is that we run perforce.

The long answer is that we run perforce with a bunch of caching servers and custom stuff in front of it and some special client wrappers. In fact there is more than one client wrapper. One of them uses git to manage a local thin client of the relevant section of the repo and sends the changes to perforce. This is the one I typically use since I get the nice tooling of git for my daily coding and then I can package all the changes up into a single perforce change later.

Google has invested a lot of infrastructure into code management and tooling to make one repo work. We've reaped a number of benefits as a result.

As others have mentioned though there are trade offs. We made the tradeoff that best suited our needs.

Re: Scaling Mercurial at Facebook

#137

Post by zaphar »

enneff wrote: post #115
X-Istence wrote: post #97

Earlier quoted context omitted.

You don't need a single repo to be able to run tests across all existing tools. If you have proper dependency management set up, you make the change, push it and a CI server goes off and builds it, then all dependent projects get rebuilt and tested...

No, but you do need a single repo if you want to make the API change and update all the dependencies in one fell swoop.

This is what facebook means when they say atomic commits.

Re: Scaling Mercurial at Facebook

#138

Post by ahomescu1 »

mushishi wrote: post #55
durin42 wrote: post #26

Earlier quoted context omitted.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

Could you elaborate on why to avoid mq?

mq keeps its patches as an internal per-repo stack. This is much weaker than git's tree branches.

Here's an example: let's say I've been working on a repo, and have 2 current mq patches called A and B pushed. There's no way for me to pop A and B, then apply another patch called D in the current repo (at least as far as I could figure out). Also, even if there were, I'd have no way of then pushing A and B on top of D. The patches only work as a linear stack (D must come on top of B).

Re: Scaling Mercurial at Facebook

#139

Post by ahomescu1 »

masklinn wrote: post #64
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

> Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid,…

Mercurial's branches are server side (learned that the hard way the first time I made a branch), git's are client side.

Re: Scaling Mercurial at Facebook

#140

Post by jordigh »

JoshTriplett wrote: post #122
masklinn wrote: post #64

Earlier quoted context omitted.

> Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid,…

Much of http://www.stationary-traveller.eu/pages/bzr-a-retrospective... applies equally well to Mercurial.

What parts? That's a very long post. I don't think the overall melancholy tone of defeat applies at all. Hg is a lively and healthy project, and the most frequently offered alternative to git with plenty of commercial backing (Facebook, Google, Atlassian, Microsoft...)
Post reply on HN