Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

31–40 of 245 posts

Re: Scaling Mercurial at Facebook

#31

Post by mixmastamyk »

JoshTriplett wrote: post #17
jordigh wrote: post #8

Earlier quoted context omitted.

> if the standard workflow doesn't include incredibly lightweight branches, I'll stick with a version control system that does. There is no real standard workflow. There are tools in place to build whatever you want. This is old, but remarkably still relevant: http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-me... If you want git-like branches, Mercurial bookmarks are close, but not quite the same. They have…

That post helped, and bookmarks might get the job done, but the mercurial culture doesn't seem to encourage them compared to branches. As a result, you tend to find a lot more separate mercurial repositories than single repos with bookmarks, and a lot more named branches with hard-to-eliminate commits as well (since even "closing" a branch doesn't really get rid of it). > hg commit --amend, hg histedit, hg record (bu…

I might have read this wrong, but safety for newbies doesn't sound all that bad to me.

Re: Scaling Mercurial at Facebook

#32

Post by Crito »

durin42 wrote: post #29
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

> An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. I'm curious - do you know of any such better examples, or is this merely a theoretical "I feel like we could do better" statement?

I've worked with more pleasant systems, but they are not public and I don't feel comfortable talking about them publicly in much detail.

Basically though, repo is good for a single project ("Android", as a collective entity) but a company looking at working with many repos will want something designed with handling many unrelated projects at once. (For example, all the repos for "Android" and all the repos or "ChromeOS" should be able to live alongside each other in this system without any developer hassle, even though they may not have anything to do with each other).

Also repo is really only one part of the solution a company should look at. Properly done, the sort of system that I am talking about is really several systems that are all basically on the same page. Your build servers should understand how to get the code they need to build something, your internal code search/browsing tools should understand where code lives, etc. Re

Furthermore, allowing different types of repos to exist alongside each other is a good idea. That way each team can individually decide if they want to use Mercurial, Git, or even SVN.. To my knowledge, the repo script is only good for many git repositories.

Re: Scaling Mercurial at Facebook

#33

Post by danielrhodes »

"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, the idea that the scaling constraints of our source control system should dictate our code structure just doesn't sit well with us."

I remember reading about their Git troubles awhile ago, and I still don't buy this argument that it is better to have one large repository. One reason modularization is important is for the precise reason they are trying to get around it: removing the ability to make large scale changes easy and thus increasing reliability.

However, my understanding is that their desire to have one large repo is reflective of their their "move fast and break things" philosophy, which means not being afraid of making large scale changes. So I would be interested in hearing how they mitigate the obvious downsides given how many people they have committing to their codebase. It seems like you would just end up having to create constraints in other ways, so which constraints end up being the lesser of two evils?

Re: Scaling Mercurial at Facebook

#34

Post by durin42 »

Crito wrote: post #32
durin42 wrote: post #29

Earlier quoted context omitted.

> An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. I'm curious - do you know of any such better examples, or is this merely a theoretical "I feel like we could do better" statement?

I've worked with more pleasant systems, but they are not public and I don't feel comfortable talking about them publicly in much detail. Basically though, repo is good for a single project ("Android", as a collective entity) but a company looking at working with many repos will want something designed with handling many unrelated projects at once. (For example, all the repos for "Android" and all the repos or "Chrome…

Would you be willing to chat about it via email? I'd love to know more, as I'm starting to look at implementing something like this and it'd be nice to reuse any work that someone has done in thinking through mistakes before I make them...

(my email is listed in my HN profile)

Re: Scaling Mercurial at Facebook

#35

Post by markcampbell »

georgemcbay wrote: post #25
Touche wrote: post #9

> We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Pray tell?

I know nothing of the code Facebook is hosting, but one fairly obvious general benefit of having things in a single repository is that you can use relative paths to link dependent projects, like if you have a directory structure such as: libs libs/mylib apps apps/myapp then myapp can reference library mylib via ../../libs/mylib If all of these things are in separate repositories, all bets are off about how their path…

There's a solution to this. Git submodules.
Post reply on HN