Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

101–110 of 245 posts

Re: Scaling Mercurial at Facebook

#101

Post by twic »

sampo wrote: post #93
VladRussian2 wrote: post #80

Earlier quoted context omitted.

>Mercurial is a DVCS my bad. 5+ years since i worked with Mercurial. Digging deep into painful memories, my Mercurial PTSD from that time is absence of in-repo branching - need to clone which is a killer for very large repo we had and no partial commits - again a killer aggravated by the above mentioned absence of in-repo branches. Both issues made working with large repo unreasonably and unnecessary hard.

I am sure Mercurial has always had (named) in-repo branches, so you were using Mercurial in strange manner.

There was a time when the developers advised using parallel clones rather than named branches, though. I never really understood why.

These days, the standard practice is probably to use bookmarks, which are more or less like Git branches.

Re: Scaling Mercurial at Facebook

#102

Post by twic »

igravious wrote: post #94
zaphar wrote: post #71

Earlier quoted context omitted.

We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywh…

What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

From what i heard, mostly Perforce.

The story i got from a Googler was that there are many separate projects in a single repository (which is normal for Perforce), and dependencies are handled by making your libraries subprojects (or whatever they're called - like subrepos in Git, essentially symlinks), rather than using an artifact-centred approach.

Re: Scaling Mercurial at Facebook

#103

Post by barkingcat »

warmwaffles wrote: post #86
ZenoArrow wrote: post #73

Earlier quoted context omitted.

There is a well used "MercurialHub", it's called BitBucket: https://bitbucket.org/ It supports Git now as well, but it was only for Mercurial use when it started.

I could have sworn bitbucket was an SVN host to start with, then patched in mercurial support and then git support later.

Bitbucket never supported svn (and still doesn't), it was created with django + uses the pure python mercurial (It was one of the major "posterchild" stories for django).

Not sure if it's still django though.

You might be thinking of http://beanstalkapp.com/ which supports svn.

Re: Scaling Mercurial at Facebook

#104

Post by kyrra »

igravious wrote: post #94
zaphar wrote: post #71

Earlier quoted context omitted.

We use one repository at google. We've found that "Removing the ability to make large scale changes easy and thus increasing reliability." isn't actually correct. As an example, most of your codebase uses an RPC library. You discover an issue with that library that requires an API change which will reduce network usage fleetwide by an order of magnitude. With a single repo it's easy to automate the API change everywh…

What version control system do you use, or is it a secret? Perforce? Git? I've seen Linus's talk that he gave about Git at the GooglePlex so perhaps you use Git. If so, how have you not run into Facebook's scaling issues?

Google runs the largest Perforce server out there.

http://www.perforce.com/blog/110607/how-do-they-do-it-google...

Re: Scaling Mercurial at Facebook

#105

Post by stormbrew »

Touche wrote: post #66
yonran wrote: post #41

Earlier quoted context omitted.

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

> * You immediately get improvements from upstream projects without having to get them manually. You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

You still immediately get regressions from separate repositories, you just don't find out until later. Possibly when you're no longer in the right mental space for dealing with it.

Re: Scaling Mercurial at Facebook

#106

Post by tonfa »

warmwaffles wrote: post #86
ZenoArrow wrote: post #73

Earlier quoted context omitted.

There is a well used "MercurialHub", it's called BitBucket: https://bitbucket.org/ It supports Git now as well, but it was only for Mercurial use when it started.

I could have sworn bitbucket was an SVN host to start with, then patched in mercurial support and then git support later.

Just re-read the original announcement, it was definitely a mercurial only hosting.

> Over the past couple of months, I've been working on creating a simple but powerful hosting service for Mercurial.

Re: Scaling Mercurial at Facebook

#107

Post by tonfa »

daleharvey wrote: post #13
durin42 wrote: post #11

Earlier quoted context omitted.

Note that Facebook is trying to scale a single large repository, not an army of slightly smaller ones. It's a very different problem, and has to be solved in a very different way.

I have seen presentations mentioning google development done as a single monolothic repository http://www.infoq.com/presentations/Development-at-Google

This monolithic repo isn't a git repo.

Re: Scaling Mercurial at Facebook

#108

Post by onestone »

durin42 wrote: post #26
JoshTriplett wrote: post #18

Earlier quoted context omitted.

They're not exactly as supported, because they're not on by default , and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

+1 for the 'hg hello' idea.

Re: Scaling Mercurial at Facebook

#109

Post by Sssnake »

cstejerean wrote: post #90
Sssnake wrote: post #85

Earlier quoted context omitted.

Git doesn't solve the issue, it suffers from it, that's the point. And as others pointed out already, mercurial is a DVCS. It just isn't as shitty as git, that's all.

Mercurial itself runs into issues way before git does, it's just that Facebook has essentially gutted a bunch of things out Mercurial in order to make the resulting thing fast (punting on things like computing status, downloading diffs, etc).

No, git runs into issues before mercurial does. Just saying the opposite of reality doesn't make it so.

Re: Scaling Mercurial at Facebook

#110

Post by moron4hire »

plorkyeran wrote: post #96
ZenoArrow wrote: post #78

Earlier quoted context omitted.

To be honest, whilst we have no way to accurately determine whether the code is a mess without a chance to see it, the most surprising line of this article (in my opinion) was that the code base was larger than the Linux kernel. I'm not seeing anything on the front end that would warrant such complexity, guessing a large chunk of the code base is server code. Would be interested in reading a summary of the components…

I suspect that the kernel is one of the only things running on Facebook's servers that they didn't write from scratch. Alexandrescu has mentioned that a 1% speedup to HHVM saves FB about $100k per year, and at that sort of scale it's pretty easy for reinventing every wheel to make sense.

right, so why can't projects like that exiat in their own repository? what keeps HHVM and Facebook tightly coupled?
Post reply on HN