Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

11–20 of 245 posts

Re: Scaling Mercurial at Facebook

#11

Post by durin42 »

xkarga00 wrote: post #4

"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project." http://www.techradar.com/news/software/how-open-source-chang... "And then Git itself wasn't working for us anymore because it wasn't scaling wh…

Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.

Re: Scaling Mercurial at Facebook

#12

Post by durin42 »

JoshTriplett wrote: post #5
ngoldbaum wrote: post #2

Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.

Re: Scaling Mercurial at Facebook

#13

Post by daleharvey »

durin42 wrote: post #11
xkarga00 wrote: post #4

"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project." http://www.techradar.com/news/software/how-open-source-chang... "And then Git itself wasn't working for us anymore because it wasn't scaling wh…

Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.

I have seen presentations mentioning google development done as a single monolothic repository

http://www.infoq.com/presentations/Development-at-Google

Re: Scaling Mercurial at Facebook

#14

Post by kisstheblade »

I wonder what they develop at facebook at that rate, and with such a huge codebase. The main site seems quite/relatively simple in functionality (and doesn't change noticeably very often). Is all that coding going towards "better" advertising :)

Re: Scaling Mercurial at Facebook

#15

Post by kyrra »

Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

From the article:

>Splitting it up would make large, atomic refactorings more difficult.

If you split the code base up into multiple projects, you need some sort of meta project to link them all together if you need to deploy it as a single entity. Doing a meta project isn't exactly easy, so it tends to be better to do a single project if everything is all tightly coupled still.

The project could be split up into smaller modules that are linked via shared common interfaces, but takes more time to maintain.

Re: Scaling Mercurial at Facebook

#16

Post by mindjiver »

At a previous job I migrated a quite large code base to git. IIRC it was at least 500k files and a couple of 10MLOCs. We had the same "scaling" issues Facebook mentioned here when trying to place all of this inside one repository. So we ended up going with submodules for this. Another idea was to perhaps enable re-use of repositories and/or disconnect old legacy code this way.

We did took a quick look at Mercurial but since lots of the upstream tools we used was using Git (linux, uboot, yocto, etc) it was an obvious choice. I seem to recall there being two hg extension that where of interest at the time (2010-ish), one to add inotify support and another to store large files outside the repository (hg-largefiles?).

Seem like Facebook's approach to the lstat(2) issue with watchman [1] is to use inotify on Linux. This has been discussed a couple of times for git as well but nothing has come of it so far [2].

[1] https://github.com/facebook/watchman [2] http://git.661346.n2.nabble.com/inotify-to-minimize-stat-cal...

Re: Scaling Mercurial at Facebook

#17

Post by JoshTriplett »

jordigh wrote: post #8
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

> if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. There is no real standard workflow. There are tools in place to build whatever you want. This is old, but remarkably still relevant: http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me... If you want git-like branches, Mercurial bookmarks are close, but not quite the same. They have…

That post helped, and bookmarks might get the job done, but the mercurial culture doesn't seem to encourage them compared to branches. As a result, you tend to find a lot more separate mercurial repositories than single repos with bookmarks, and a lot more named branches with hard-to-eliminate commits as well (since even "closing" a branch doesn't really get rid of it).

> hg commit --amend, hg histedit, hg record (but hg crecord is much nicer).

> They're built-in, but you have to flip them on the last two with a config switch.

And that's the fundamental attitude that I can't stand in mercurial's culture: "careful, you might accidentally throw away that information you specifically wanted to throw away". Those commands aren't available by default because the culture doesn't encourage that workflow, so even if you use them for your own personal workflow you'll tend to find a lot more junk commits and forever-remembered branch names in the average mercurial repository.

Re: Scaling Mercurial at Facebook

#18

Post by JoshTriplett »

durin42 wrote: post #12
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.

They're not exactly as supported, because they're not on by default, and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

Re: Scaling Mercurial at Facebook

#20

Post by dudus »

Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

On the other hand the Mercurial modularity was key for the migration.
Post reply on HN