Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

41–50 of 245 posts

Re: Scaling Mercurial at Facebook

#41

Post by yonran »

Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages:

* You immediately get improvements from upstream projects without having to get them manually.

* You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using.

* Easy API refactoring. You don’t have to worry about coordinating version number bumps across different repositories/dependency manifests when you make major changes to inter-project APIs. With a monolithic repository, you fix all callers of an API using a single code commit. No need to edit the version numbers in your pom files.

* Low cost to split a project into multiple separate projects. With multiple repositories or version numbers, you are reluctant to create new projects because the APIs will forever be harder to refactor (since you will have to worry about version numbers).

* No diamond dependency problem of 2 dependencies using a different version of a base project. Everyone is using the same version of base.

With a monolithic repository+build system, upstream callers are responsible to never make a commit that breaks downstream callers. I feel like it’s similar to the question of optimal currency areas. If your organization is growing in lock-step, then you can all happily share a single gold-standard repository with little friction. But if you can’t trust your upstream projects, you introduce versioning between the projects and have to deal with the mental burden of wondering whether to upgrade to the newest upstream project and whether you’re actually running the latest code.

Edit: added a couple more.

Re: Scaling Mercurial at Facebook

#42

Post by Timmmmbob »

VladRussian2 wrote: post #28
LukeHoersten wrote: post #3

I love the Mercurial community. We use Mercurial at work and I'm able to get instant support in IRC for any issue we have with an awesome signal/noise ratio. I'm glad Facebook is contributing back so much as well. My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. All anecdotal observati…

>My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice…

That makes no sense. Facebook specifically said they might have chosen git! They had the same choices as any developer except that a) they don't care about github, and b) they wanted to modify the VCS to support massive scale. Apparently out of git and mercurial, mercurial is easier to extend.

Re: Scaling Mercurial at Facebook

#43

Post by Uchikoma »

JoshTriplett wrote: post #5
ngoldbaum wrote: post #2

Mercurial has seriously improved over the past couple of years. If you tried mercurial a few years ago and were scared away due to speed or functionality issues, you might want to give it another shot.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Let me translate: "if this tool is not exactly like Git, I will not use it".

Re: Scaling Mercurial at Facebook

#44

Post by Timmmmbob »

durin42 wrote: post #11
xkarga00 wrote: post #4

"Our engineers were comfortable with Git and we preferred to stay with a familiar tool, so we took a long, hard look at improving it to work at scale. After much deliberation, we concluded that Git's internals would be difficult to work with for an ambitious scaling project." http://www.techradar.com/news/software/how-open-source-chang... "And then Git itself wasn't working for us anymore because it wasn't scaling wh…

Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.

Android is only split up into smaller repositories because git can't handle having it all in one.

Re: Scaling Mercurial at Facebook

#45

Post by VladRussian2 »

Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

>With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team.

>Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every object in the DAG), even ignoring throughput.

if you use Git the same way you'd use Perforce or Mercurial - one central server handling everything ... you'd better stay with Perforce or Mercurial. Corporate dev process is usually built around notion of "central" repo (and they equate it with "master") - just replacing Perforce server with Git server wouldn't change anything.

Re: Scaling Mercurial at Facebook

#46

Post by Uchikoma »

VladRussian2 wrote: post #28
LukeHoersten wrote: post #3

I love the Mercurial community. We use Mercurial at work and I'm able to get instant support in IRC for any issue we have with an awesome signal/noise ratio. I'm glad Facebook is contributing back so much as well. My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. All anecdotal observati…

>My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice…

90% of people choose Git because everyone else uses it. Noone has been fired for buying IBM.

Re: Scaling Mercurial at Facebook

#47

Post by wtetzner »

k_bx wrote: post #21
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Why do you think mercurial branches are not lightweight? Which operation (create, close, push, pull) on branches is noticeably slower than git's?

I don't think he was talking about performance. I think he meant lightweight in the sense that a branch (in git) is just a pointer to a commit.

It's conceptually lightweight.

Re: Scaling Mercurial at Facebook

#48

Post by bostik »

markcampbell wrote: post #35
georgemcbay wrote: post #25

Earlier quoted context omitted.

I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as: libs libs/mylib apps apps/myapp then myapp can reference library mylib via ../../libs/mylib If all of these things are in separate repositories, all bets are off about how their path…

There's a solution to this. Git submodules.

Please tell me if there's a way to make git-grep (or git-log for that matter) work between submodules.

While I may not personally like behemoth repositories, I do see why people would choose to endure them. They do have their advantages, even if they all eventually end up looking something like BSD ports. (FYI: I happen to like the extremely modular one-git-repo-per-component approach as used in Mer. Hell, I did an early implementation of the OBS+git -based autobuild system back in 2009!)

Re: Scaling Mercurial at Facebook

#49

Post by natrius »

durin42 wrote: post #30
natrius wrote: post #19

Branches are my biggest pain point in Mercurial. What branching workflow does Facebook use?

In what way? Have you looked at bookmarks?

I have. I'm wondering if they use bookmarks or named branches, and what strategies they've arrived at to work with either.

Re: Scaling Mercurial at Facebook

#50

Post by natrius »

k_bx wrote: post #22
natrius wrote: post #19

Branches are my biggest pain point in Mercurial. What branching workflow does Facebook use?

Why not use standard mercurial branches? At least you're able to say in which branch commit was done and draw a clean history for them.

I do use standard Mercurial branches, and I vastly prefer Git's model. I ask because there might be a better way to use them that I've overlooked.
Post reply on HN