Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

121–130 of 245 posts

Re: Scaling Mercurial at Facebook

#121

Post by JoshTriplett »

mixmastamyk wrote: post #31
JoshTriplett wrote: post #17

Earlier quoted context omitted.

That post helped, and bookmarks might get the job done, but the mercurial culture doesn't seem to encourage them compared to branches. As a result, you tend to find a lot more separate mercurial repositories than single repos with bookmarks, and a lot more named branches with hard-to-eliminate commits as well (since even "closing" a branch doesn't really get rid of it). > hg commit --amend, hg histedit, hg record (bu…

I might have read this wrong, but safety for newbies doesn't sound all that bad to me.

History editing in git is not unsafe: because it's a part of the default set of tools, git offers functionality like the reflog to keep track of old history locally and avoid losing it. It's extremely difficult to permanently lose content in a git repository even using the history editing commands; it's always just a "git reflog" away.

Re: Scaling Mercurial at Facebook

#122

Post by JoshTriplett »

masklinn wrote: post #64
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

> Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid,…

Much of http://www.stationary-traveller.eu/pages/bzr-a-retrospective... applies equally well to Mercurial.

Re: Scaling Mercurial at Facebook

#123

Post by JoshTriplett »

berdario wrote: post #84
masklinn wrote: post #64

Earlier quoted context omitted.

> Does the standard branch workflow still expect you to have a separate repository and directory per branch? You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo). > I don't care about plugins That's stupid,…

Furthermore, he's probably confusing mercurial with something that don't even exist bazaar expects you to have a separate working copy (directory) for each branch, but you can have multiple branches stored in the same repository without any problem (I just wished that git people actually knew how do other tools work... but, alas! Now it's too late for underdogs like bazaar or darcs to catch up)

> bazaar expects you to have a separate working copy (directory) for each branch, but you can have multiple branches stored in the same repository without any problem

Those are functionally equivalent, in that they both mean branching is not instantaneous.

Re: Scaling Mercurial at Facebook

#124

Post by JoshTriplett »

wtetzner wrote: post #47
k_bx wrote: post #21

Earlier quoted context omitted.

Why do you think mercurial branches are not lightweight? Which operation (create, close, push, pull) on branches is noticeably slower than git's?

I don't think he was talking about performance. I think he meant lightweight in the sense that a branch (in git) is just a pointer to a commit. It's conceptually lightweight.

Both: it's conceptually lightweight, and because of that "pointer to a commit" structure it's also effectively free to create and move around branches.

Re: Scaling Mercurial at Facebook

#125

Post by JoshTriplett »

Uchikoma wrote: post #43
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Let me translate: "if this tool is not exactly like Git, I will not use it".

Not exactly, but close. Switching away from Git would require a tool that is both compellingly better in some key way as well as not worse for any existing workflow. I have yet to see any other version control system that meets both of those criteria, even leaving aside the network effects of using the most popular system.

Re: Scaling Mercurial at Facebook

#126

Post by rbehrends »

durin42 wrote: post #26
JoshTriplett wrote: post #18

Earlier quoted context omitted.

They're not exactly as supported, because they're not on by default , and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

I have wondered why they're called "extensions" rather than "modules", which I think would characterize their actual use better.

Re: Scaling Mercurial at Facebook

#127

Post by eddieh »

dubcanada wrote: post #120
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine. But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

WebKit uses Subversion (http://www.webkit.org/building/checkout.html). You can use Git with the WebKit repo, but it's git-svn doing the work...

Re: Scaling Mercurial at Facebook

#128

Post by greatsuccess »

Partitioning is the answer. In a repo that size 99% of the history is useless to anyone. You wouldnt manage a database like this so why force SCM down this path?

If they used git with say only the last year of history in it they would be having zero issues.

Re: Scaling Mercurial at Facebook

#129

Post by rbehrends »

JoshTriplett wrote: post #123
berdario wrote: post #84

Earlier quoted context omitted.

Furthermore, he's probably confusing mercurial with something that don't even exist bazaar expects you to have a separate working copy (directory) for each branch, but you can have multiple branches stored in the same repository without any problem (I just wished that git people actually knew how do other tools work... but, alas! Now it's too late for underdogs like bazaar or darcs to catch up)

> bazaar expects you to have a separate working copy (directory) for each branch, but you can have multiple branches stored in the same repository without any problem Those are functionally equivalent, in that they both mean branching is not instantaneous.

Those are functionally equivalent, in that they both mean branching is not instantaneous.

They are not functionally equivalent. A "bzr branch" with a shared repository will only populate the working tree and does not have to duplicate the repository. It is functionally closer to "git-new-workdir" than "git clone" or "git branch".

To have instant branching in Bazaar, you can use co-located branches.

Branches with their own directories exist for the use case where the accumulated cost for rebuilds after branch switches is more costly than populating a new directory with a checkout or if you need two checkouts in parallel. They also exist in Bazaar for simulating a workflow with a central server and no local repository data.

Re: Scaling Mercurial at Facebook

#130

Post by moron4hire »

enneff wrote: post #117
moron4hire wrote: post #27

> "Our code base has grown organically and its internal dependencies are very complex." That's a polite way of saying "we write shitty code without any sort of plan." > "Splitting it up would make large, atomic refactorings more difficult" Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings. And what, exactly, is the meaning of these graphs? This is leading me to b…

> Modularity tends to obviate the need for large, atomic refactorings. "Tends to". But when you're dealing with code at Facebook's scale, things that "tend not to happen" actually happen quite a lot. In fact, you must plan for them as a matter of course. So yes, modularity is great, and I because I'm a nice guy I assume Facebook aren't a pack of idiots and that they're writing nice modular code. But even if that's th…

Perhaps I don't understand the whole situation here. I hear "all of our code is in one repository" and I think "GMail and Google Maps are in the same repository, in the same repository with GoLang, in the same repository with AdWords."

The more I think about it, the more I think your post reveals a lack of maturity in our industry that lends credence to the pro-engineering-licensing argument that I've argued against many times on my own. That everyone can be so cavalier about this topic.

Because the fact that your companies are so large is EXACTLY why it makes no flipping sense that you're running gigantorepository. You have so many products, so many projects going on, that I just really have a hard time believing that it was disciplined software development that led to all of your code being so interdependent.

But the part that started getting under my skin was the fact that we aren't talking about Bob's Local Software Consultancy here. We're talking about two companies that touch the lives of hundreds of millions, perhaps even billions of people in the world.

If OpenStreetMap doesn't have their code in the same repository as Postgres, Linux, and DuckDuckGo, then there is no excuse for the Facebook Android App to be in the same repository as HHVM.

Post reply on HN