Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

51–60 of 245 posts

Re: Scaling Mercurial at Facebook

#51

Post by ariwilson »

Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

How do you import code from one repo into another?

This is the fundamental problem Facebook (and Google, Amazon, etc) are all trying to solve: how to share code company-wide. Sub-repos and making your tools hack around the repos doesn't solve that problem.

Re: Scaling Mercurial at Facebook

#52

Post by nilved »

markcampbell wrote: post #35
georgemcbay wrote: post #25

Earlier quoted context omitted.

I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as: libs libs/mylib apps apps/myapp then myapp can reference library mylib via ../../libs/mylib If all of these things are in separate repositories, all bets are off about how their path…

There's a solution to this. Git submodules.

git submodules don't fix anything except maybe a good day.

Re: Scaling Mercurial at Facebook

#53

Post by Crito »

ariwilson wrote: post #51
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

How do you import code from one repo into another? This is the fundamental problem Facebook (and Google, Amazon, etc) are all trying to solve: how to share code company-wide. Sub-repos and making your tools hack around the repos doesn't solve that problem.

I think ultimately, past the scale that you can get with Perforce, that is a problem waiting to be solved by the next generation of VCS.

You can get all your many repos in one place, with all of your internal systems handling them nicely, but moving one file from one repo to another, in a way that history remains clear, is something of an unsolved problem.

Re: Scaling Mercurial at Facebook

#54

Post by cstejerean »

ariwilson wrote: post #51
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

How do you import code from one repo into another? This is the fundamental problem Facebook (and Google, Amazon, etc) are all trying to solve: how to share code company-wide. Sub-repos and making your tools hack around the repos doesn't solve that problem.

You publish packages/libraries/gems/jars/whatever your language calls them, and a package manager. Needing to combine the actual source trees into a single repo like submodules or subtrees allow you to should be for really rare cases.

Re: Scaling Mercurial at Facebook

#55

Post by mushishi »

durin42 wrote: post #26
JoshTriplett wrote: post #18

Earlier quoted context omitted.

They're not exactly as supported, because they're not on by default , and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

Could you elaborate on why to avoid mq?

Re: Scaling Mercurial at Facebook

#56

Post by Crito »

VladRussian2 wrote: post #45
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

>With perforce, you can reasonably expect to run into this brick wall somewhere in the neighborhood of terabytes of metadata and dozens of transactions per second. That changes depending on what sort of beastly hardware you are willing to throw at your version control team. >Git of course hits a brick wall much sooner, somewhere around single-digit gigabytes of data (depending heavily on the average size of every obj…

That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

The problem is "how do you grow even further?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos.

Thankfully very few companies need to worry about this problem right now. A single Mercurial repo is apparently scaling for Facebook currently, and few companies will manage to stress Perforce.

Re: Scaling Mercurial at Facebook

#57

Post by justinmk »

Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

> Facebook's problem is that they were trying to scale with Git improperly.

Here's the Feb 2012 thread with Facebook's Joshua Redstone regarding their experiments with git:

http://comments.gmane.org/gmane.comp.version-control.git/189...

In August 2012, they hinted they were near a solution:

http://www.quora.com/Facebook-Engineering/Has-Facebook-solve...

...and that teaser was just today updated with the remark, "Our Mercurial team has just posted the goods! Our problems are solved, mainly by not using Git."

:/

Re: Scaling Mercurial at Facebook

#58

Post by gecko »

Uchikoma wrote: post #43
JoshTriplett wrote: post #5

Earlier quoted context omitted.

Does the standard branch workflow still expect you to have a separate repository and directory per branch? I don't care about plugins, here; if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. Likewise, does the standard workflow still intentionally make it painful to rearrange changes in your local repository to construct a series of patches?…

Let me translate: "if this tool is not exactly like Git, I will not use it".

That sentiment comprises a truly mindblowing number of "bug" reports that hit the Mercurial mailing list every month. I honestly wonder sometimes if I'm simply forgetting how CVS users did the same thing when Subversion shipped and I forgot about it/missed it, or what.

Re: Scaling Mercurial at Facebook

#59

Post by Crito »

cstejerean wrote: post #54
ariwilson wrote: post #51

Earlier quoted context omitted.

How do you import code from one repo into another? This is the fundamental problem Facebook (and Google, Amazon, etc) are all trying to solve: how to share code company-wide. Sub-repos and making your tools hack around the repos doesn't solve that problem.

You publish packages/libraries/gems/jars/whatever your language calls them, and a package manager. Needing to combine the actual source trees into a single repo like submodules or subtrees allow you to should be for really rare cases.

That reduces the problem, though it doesn't eliminate it. You'll still run into rough spots when you go to rearrange what files are in what packages.

Say one aspect of a package suddenly starts to grow quite fast and take on a life of its own, and you would like to split it into its own dedicated package (and therefore, it's own repo). How do you do that without loosing the history of those files?

It's possible with git.. but it isn't exactly what I would call straightforward. With a one-repo system you simply move those files to their new location, just like any other file move/rename.

Re: Scaling Mercurial at Facebook

#60

Post by Touche »

kyrra wrote: post #15
Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

From the article: >Splitting it up would make large, atomic refactorings more difficult. If you split the code base up into multiple projects, you need some sort of meta project to link them all together if you need to deploy it as a single entity. Doing a meta project isn't exactly easy, so it tends to be better to do a single project if everything is all tightly coupled still. The project could be split up into sma…

What is a wrong with submodules or subtree for a meta project? Or better yet, package management.
Post reply on HN