Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

61–70 of 245 posts

Re: Scaling Mercurial at Facebook

#61

Post by VladRussian2 »

Crito wrote: post #56
VladRussian2 wrote: post #45

Earlier quoted context omitted.

>With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team. >Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every obj…

That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go. The problem is "how do you grow even further ?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos. Thankfully very few companies need to worry about this prob…

> If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

that is the problem - single logical repo in Perforce/Mercurial does mean single physical repo. Which obviously causes issues at very large scale. Even at normal enterprise scale :)

Git solves it through kind of distributed scaling where many operations can be performed on local physical repos without ever hitting the central. With centralized solutions, like Perforce, Mercurial, etc... most of the operations are performed on central server and your ability to scale vertically hits the ceiling pretty soon.

Compare the simplest case - each dev updating his local workspace/repo would hit the central server in Perforce where is with Git you can (and normally would have) a small set of downstream repos which the updates would be propagated/distributed through. You can branch left and right in Git in your local and team's repos without "master" repo and "central" server involved (while all your branch activity in Perforce is happening right on the central server in the central repo). The same happens for synchronizing your work inside the team - no need to hit the central. Etc...

Re: Scaling Mercurial at Facebook

#62

Post by jpace121 »

durin42 wrote: post #26
JoshTriplett wrote: post #18

Earlier quoted context omitted.

They're not exactly as supported, because they're not on by default , and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

In response to your last sentence, yes.

Re: Scaling Mercurial at Facebook

#63

Post by boyter »

mindjiver wrote: post #38
ed_blackburn wrote: post #37

I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.

I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.

I saw/heard that as well (hopefully someone can come up with the link). Things like Windows/Office apparently are in a custom perforce.

I believe that they do use TFS on a lot of the internal projects though and that Visual Studio is now done in TFS.

Re: Scaling Mercurial at Facebook

#64

Post by masklinn »

JoshTriplett wrote: post #5
ngoldbaum wrote: post #2

Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

> Does the standard branch workflow still expect you to have a separate repository and directory per branch?

You're probably confusing mercurial with bazaar, mercurial has always had branches (though they're not quite the same as git, mercurial's bookmarks are more closely related to git branches) and anonymous heads (contrary to git, an unnamed head is not stuck in limbo).

> I don't care about plugins

That's stupid, mercurial is very much about plugins: there are dozens of official plugins shipped in a standard mercurial install.

> the standard workflow

there is no such thing.

> Does Mercurial provide built-in commands equivalent to "commit --amend", "rebase -i", and "add -p"?

All of them are provided in the base install, you just have to enable the corresponding extensions.

Re: Scaling Mercurial at Facebook

#65

Post by masklinn »

mushishi wrote: post #55
durin42 wrote: post #26

Earlier quoted context omitted.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

Could you elaborate on why to avoid mq?

It's a very powerful tool, but also a complex one, and one which brings in a bunch of new commands. There are a bunch of extension which provide nice UIs to various subsets of mq's full power, and are much simpler.

If you need mq, use mq, but if you just need to e.g. do some history edition, use histedit.

Re: Scaling Mercurial at Facebook

#66

Post by Touche »

yonran wrote: post #41
Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

> * You immediately get improvements from upstream projects without having to get them manually.

You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

Re: Scaling Mercurial at Facebook

#67

Post by ash »

Crito wrote: post #59
cstejerean wrote: post #54

Earlier quoted context omitted.

You publish packages/libraries/gems/jars/whatever your language calls them, and a package manager. Needing to combine the actual source trees into a single repo like submodules or subtrees allow you to should be for really rare cases.

That reduces the problem, though it doesn't eliminate it. You'll still run into rough spots when you go to rearrange what files are in what packages. Say one aspect of a package suddenly starts to grow quite fast and take on a life of its own, and you would like to split it into its own dedicated package (and therefore, it's own repo). How do you do that without loosing the history of those files? It's possible with…

> How do you do that without loosing the history of those files?

I don't know how you did it, but I found `git subtree split` to be an easy solution for extracting a directory into its own repo:

https://github.com/git/git/blob/master/contrib/subtree/git-s...

(It's a shame git-subtree it's still in contrib/, should really be enabled by default. It can also work as git-submodule replacement in some cases, by the way.)

Re: Scaling Mercurial at Facebook

#68

Post by masklinn »

natrius wrote: post #50
k_bx wrote: post #22

Earlier quoted context omitted.

Why not use standard mercurial branches? At least you're able to say in which branch commit was done and draw a clean history for them.

I do use standard Mercurial branches, and I vastly prefer Git's model. I ask because there might be a better way to use them that I've overlooked.

Mercurial's equivalent to git branches (movable pointer to a commit rather than embedded commit metadata) is bookmarks.

Re: Scaling Mercurial at Facebook

#69

Post by Crito »

ash wrote: post #67
Crito wrote: post #59

Earlier quoted context omitted.

That reduces the problem, though it doesn't eliminate it. You'll still run into rough spots when you go to rearrange what files are in what packages. Say one aspect of a package suddenly starts to grow quite fast and take on a life of its own, and you would like to split it into its own dedicated package (and therefore, it's own repo). How do you do that without loosing the history of those files? It's possible with…

> How do you do that without loosing the history of those files? I don't know how you did it, but I found `git subtree split` to be an easy solution for extracting a directory into its own repo: https://github.com/git/git/blob/master/contrib/subtree/git-s... (It's a shame git-subtree it's still in contrib/, should really be enabled by default. It can also work as git-submodule replacement in some cases, by the way.)

Even if it isn't neatly in it's own tree, you can get away with rewriting commits and some tree objects. It's more complicated, but it can be made to work.

Either isn't as nice as `p4 move ...` though.

Re: Scaling Mercurial at Facebook

#70

Post by pnathan »

durin42 wrote: post #29
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

> An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. I'm curious - do you know of any such better examples, or is this merely a theoretical "I feel like we could do better" statement?

I participated in creating and open sourcing Guestrepo for hg. It is a very nice multirepo infrastructure, in my humble opinion. ;-)

https://bitbucket.org/selinc/guestrepo

Post reply on HN