Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

91–100 of 245 posts

Re: Scaling Mercurial at Facebook

#91

Post by jordigh »

mushishi wrote: post #55
durin42 wrote: post #26

Earlier quoted context omitted.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

Could you elaborate on why to avoid mq?

Some people like it, but my basic complaint with it is that it introduces a new concept, the "patch", that is basically a crippled commit. When a patch is not a commit, you get rejected diff hunks when it fails to apply, and it can easily forget its history.

An MQ patch is basically a commit that doesn't know how to merge and doesn't keep backups. It's way too easy to make a mistake and lose work. I consider MQ one of hg's youthful mistakes before tools like histedit, rebase, and Evolve came to exist.

Re: Scaling Mercurial at Facebook

#92

Post by yonran »

Touche wrote: post #66
yonran wrote: post #41

Earlier quoted context omitted.

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

> * You immediately get improvements from upstream projects without having to get them manually. You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

You’re right, there are disadvantages to a single codebase and builds at trunk. You pick up regressions, and it is difficult for your team to develop on its own while holding everything else constant.

But I don’t think the speed of receiving fixes is the essence of, or even primarily the source of, all my headaches with versions. The problem with mixing and matching versions within an organization is the enormous complexity that it introduces. Perhaps your downstream coworkers are still using an old version of your project, so they don’t want you to refactor their use of your API. Or perhaps you forgot to update your required upstream dependency when using a new function from a library, and your coworker’s program crashed because they’re still using the old dependency. Or perhaps someone forgot to bump the major version number when changing API or behavior, causing a previously built downstream project that is linking against the new upstream project to crash.

Now, these problems are all solvable if you and your coworkers are very disciplined in updating your version numbers and your required dependency versions. But it means that you constantly have to be aware of what APIs you export and what versions of APIs you are calling. You constantly have to edit the project manifests to bump version numbers. You must think about whether your changes will be major or minor. You carefully read the Changelist before using the newest upstream projects. It is a mental burden.

Contrast this to a monolithic codebase and build system. There are no version numbers in the dependency manifests to other projects in the company. If you want to change an API, you are responsible for fixing all the downstream users (rather than the other way around). Making a new project adds little mental overhead. If there is no impedance mismatch between the different teams of your company, it can make life much easier.

Re: Scaling Mercurial at Facebook

#93

Post by sampo »

VladRussian2 wrote: post #80
ngoldbaum wrote: post #72

Earlier quoted context omitted.

Mercurial is a DVCS, just like git. It has no concept of a central server.

>Mercurial is a DVCS my bad. 5+ years since i worked with Mercurial. Digging deep into painful memories, my Mercurial PTSD from that time is absence of in-repo branching - need to clone which is a killer for very large repo we had and no partial commits - again a killer aggravated by the above mentioned absence of in-repo branches. Both issues made working with large repo unreasonably and unnecessary hard.

I am sure Mercurial has always had (named) in-repo branches, so you were using Mercurial in strange manner.

Re: Scaling Mercurial at Facebook

#94

Post by igravious »

zaphar wrote: post #71
danielrhodes wrote: post #33

"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, t…

We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywh…

What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

Re: Scaling Mercurial at Facebook

#95

Post by adamnemecek »

mindjiver wrote: post #38
ed_blackburn wrote: post #37

I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.

I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.

This is correct.

http://programmers.stackexchange.com/a/85891

Re: Scaling Mercurial at Facebook

#96

Post by plorkyeran »

ZenoArrow wrote: post #78
moron4hire wrote: post #27

> "Our code base has grown organically and its internal dependencies are very complex." That's a polite way of saying "we write shitty code without any sort of plan." > "Splitting it up would make large, atomic refactorings more difficult" Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings. And what, exactly, is the meaning of these graphs? This is leading me to b…

To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components…

I suspect that the kernel is one of the only things running on Facebook's servers that they didn't write from scratch. Alexandrescu has mentioned that a 1% speedup to HHVM saves FB about $100k per year, and at that sort of scale it's pretty easy for reinventing every wheel to make sense.

Re: Scaling Mercurial at Facebook

#97

Post by X-Istence »

zaphar wrote: post #71
danielrhodes wrote: post #33

"We could have spent a lot of time making it more modular in a way that would be friendly to a source control tool, but there are a number of benefits to using a single repository. Even at our current scale, we often make large changes throughout our code base, and having a single repository is useful for continuous modernization. Splitting it up would make large, atomic refactorings more difficult. On top of that, t…

We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywh…

You don't need a single repo to be able to run tests across all existing tools. If you have proper dependency management set up, you make the change, push it and a CI server goes off and builds it, then all dependent projects get rebuilt and tested...

Re: Scaling Mercurial at Facebook

#98

Post by igravious »

ZenoArrow wrote: post #78
moron4hire wrote: post #27

> "Our code base has grown organically and its internal dependencies are very complex." That's a polite way of saying "we write shitty code without any sort of plan." > "Splitting it up would make large, atomic refactorings more difficult" Actually, it's the other way around. Modularity tends to obviate the need for large, atomic refactorings. And what, exactly, is the meaning of these graphs? This is leading me to b…

To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components…

This rather surprised me as well. I tend to think of the Linux kernel as one of the larger single code-bases out there. Am I wrong?

Re: Scaling Mercurial at Facebook

#99

Post by robaato »

boyter wrote: post #63
mindjiver wrote: post #38

Earlier quoted context omitted.

I read somewhere(?) that they where running Perforce with some internal tooling built on top of it.

I saw/heard that as well (hopefully someone can come up with the link). Things like Windows/Office apparently are in a custom perforce. I believe that they do use TFS on a lot of the internal projects though and that Visual Studio is now done in TFS.

The following is a presentation in 2008 by Richard Erwin of Microsoft at the BCS CMSG (Configuration Management Specialist Group):

http://bcs-cmsg.org.uk/events/e20081124/2008-11-24-agile-scm...

Lots of good stuff there, but a key one is slide 8. In the question session Richard confirmed that Source Depot (the custom version of Perforce) was at that time still used for the source management although TFS was used for bug tracking etc for Office and Windows.

Don't know what has happened in the intervening years...

Re: Scaling Mercurial at Facebook

#100

Post by twic »

pnathan wrote: post #70
durin42 wrote: post #29

Earlier quoted context omitted.

> An example of building multi-repo infrastructure for large projects with git is Android's repo: http://en.wikipedia.org/wiki/Repo_(script) Repo is just one example though; other, better, solutions are very possible. I'm curious - do you know of any such better examples, or is this merely a theoretical "I feel like we could do better" statement?

I participated in creating and open sourcing Guestrepo for hg. It is a very nice multirepo infrastructure, in my humble opinion. ;-) https://bitbucket.org/selinc/guestrepo

+1. I was reading the Mercurial mailing list while the Guestrepo design was being hashed out, and i was really impressed: the goals and mechanism were carefully considered. I haven't actually used it, so i can't vouch for the implementation!
Post reply on HN