Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

151–160 of 245 posts

Re: Scaling Mercurial at Facebook

#151

Post by dgesang »

rbehrends wrote: post #126
durin42 wrote: post #26

Earlier quoted context omitted.

No, they're exactly as supported . What I meant by that was that we promise to not break them, ever, to keep the output formats stable, and accept bug reports for them. That doesn't necessarily mean it's something we'll always recommend (eg mq isn't something I'd recommend for a new user, rebase/histedit/amend are way better and always will be.) We don't turn them on by default for two reasons: newbie users not shoot…

I have wondered why they're called "extensions" rather than "modules", which I think would characterize their actual use better.

There are only two hard problems in computer science: cache invalidation and naming things. - Phil Karlton

Re: Scaling Mercurial at Facebook

#152

Post by thrownaway2424 »

moron4hire wrote: post #130
enneff wrote: post #117

Earlier quoted context omitted.

> Modularity tends to obviate the need for large, atomic refactorings. "Tends to". But when you're dealing with code at Facebook's scale, things that "tend not to happen" actually happen quite a lot. In fact, you must plan for them as a matter of course. So yes, modularity is great, and I because I'm a nice guy I assume Facebook aren't a pack of idiots and that they're writing nice modular code. But even if that's th…

Perhaps I don't understand the whole situation here. I hear "all of our code is in one repository" and I think "GMail and Google Maps are in the same repository, in the same repository with GoLang, in the same repository with AdWords." The more I think about it, the more I think your post reveals a lack of maturity in our industry that lends credence to the pro-engineering-licensing argument that I've argued against…

The fact that Google is one of the largest, most successful software companies in history and you are arguing on the internet using the handle "moron4hire" just about sums up the merits of your position.

Re: Scaling Mercurial at Facebook

#153

Post by dgesang »

JoshTriplett wrote: post #18
durin42 wrote: post #12

Earlier quoted context omitted.

Note that extensions that ship with Mercurial are exactly as supported as the tool itself, and come with the same kinds of backwards compatibility guarantees. It's not worth worrying about rebase, histedit, and record being extensions - if you want them, turn them on and use them.

They're not exactly as supported, because they're not on by default , and they're not the thing everyone in the Mercurial culture tells you to use as the obvious solution to problems. Version control is as much about what other people do as what you do, and what other people do tends to align most closely with the defaults and what the tools encourage.

> They're not exactly as supported, because they're not on by default

I really don't get this. In almost any software there are features that are not 'on by default', but it's still implemented and supported (except explicitly stated otherwise). How can this be a reason to not use the functionality when you need it?

Re: Scaling Mercurial at Facebook

#154

Post by randartie »

dubcanada wrote: post #120
Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine. But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

It's a bold assumption that those open source projects are larger than facebook, a company which employs 2000+ engineers.

webkit and chromium have about 6 or 7 million lines of code and Facebook 6 months ago had 62 million lines of code and > 1000 commits a day.

http://www.quora.com/Facebook-Engineering/How-many-lines-of-...

Also, there's a difference between those open source projects using large repos with git and Facebook wanting to increase developer efficiency. Developers sitting around waiting for a rebase doesn't really pay off, where as an open source project can get away with it.

Re: Scaling Mercurial at Facebook

#155

Post by ibrahima »

Crito wrote: post #24

I wrote a comment about the scaling of repositories (and specifically Facebook's issues) a few days ago that was wiped out by the HN crash, but I've managed to recover it from HNSearch: Facebook's problem is that they were trying to scale with Git improperly. With conventional CVS[sic] systems like Perforce, you can scale a single repo nearly as large as any company will need. Emphasis on that nearly. At a certain po…

I get this impression every time I see a "look what neat scalability thing we did" post from Facebook Engineering. It's great that they're able to achieve such technical feats, but they refuse to acknowledge that maybe they're using the technology wrong. I'm reminded of the time they hacked the Dalvik VM on Android because apparently they had too many method names for the Dalvik VM to handle. http://jaxenter.com/facebook-s-completely-insane-dalvik-hack...

I mean, they run into a resource allocation problem for an application that is essentially a glorified web view, and they think "how can I hack up the VM to bypass this limitation?" Sounds like insanity to me.

Re: Scaling Mercurial at Facebook

#156

Post by randartie »

agwa wrote: post #39

For some context of why Facebook choose Hg over Git, here's the mailing list thread where Facebook initially reached out to the Git developers: http://thread.gmane.org/gmane.comp.version-control.git/18977...

The discussion appears deleted on that thread

Re: Scaling Mercurial at Facebook

#157

Post by ibrahima »

Sssnake wrote: post #109
cstejerean wrote: post #90

Earlier quoted context omitted.

Mercurial itself runs into issues way before git does, it's just that Facebook has essentially gutted a bunch of things out Mercurial in order to make the resulting thing fast (punting on things like computing status, downloading diffs, etc).

No, git runs into issues before mercurial does. Just saying the opposite of reality doesn't make it so.

Did you even read the post? Stock git is significantly faster than stock mercurial, it's only after they hack it up that they make it faster.

Re: Scaling Mercurial at Facebook

#158

Post by thrownaway2424 »

sandGorgon wrote: post #148
zaphar wrote: post #136

Earlier quoted context omitted.

This question is actually really complicated to answer. The short answer is that we run perforce. The long answer is that we run perforce with a bunch of caching servers and custom stuff in front of it and some special client wrappers. In fact there is more than one client wrapper. One of them uses git to manage a local thin client of the relevant section of the repo and sends the changes to perforce. This is the one…

Interesting, has Google made the client parts open source? It would be great if someone used that to write a high performance git server. Could they?

There is no such thing as a high performance git anything. People at Google who use git (I used to be one of them) suffer from far worse performance than the people who just use perforce as such. In particular, the cost of a git gc, which invariably occurs right when you're on the verge of an epic hack, or fighting a gigantic outage, is unreasonable and perforce has no analogous periodic process.

Re: Scaling Mercurial at Facebook

#159

Post by thrownaway2424 »

Touche wrote: post #66
yonran wrote: post #41

Earlier quoted context omitted.

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

> * You immediately get improvements from upstream projects without having to get them manually. You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

Immediate regressions are good! If someone at Google breaks my code, I will know within half an hour at the latest and I will tell them to go fix it or just revert their changes myself. Immediate regressions also go perfectly with daily (or hourly!) releases. If there's a performance problem it will be identified early and I will only have thousands of changes to investigate instead of tens of millions.

Imagine if I had a regression and I had to go to the other team saying "We just upgraded from the Foo you released 2 years ago to the one from last year and the performance sucks. Help!" I would not get any help. However I get plenty of support when I go to Foo-team to tell them that my Foo-per-second is 10% worse in the noon release compared to the midnight release.

Having artifacts and stable interfaces and library releases and all that is very ivory tower hocus pocus stuff. In practice instant integration is better.

Re: Scaling Mercurial at Facebook

#160

Post by Shamanmuni »

Uchikoma wrote: post #46
VladRussian2 wrote: post #28

Earlier quoted context omitted.

>My suspicion is that open source projects tend towards Git because of GitHub but I think a lot of companies who don't have the option of external code hosting lean towards Mercurial. GitHub is a consequence not the cause (ponder for a moment why there is no MercurialHub...) It is about ability to choose best source control tool for multi-versioned distributed concurrent development. Open source devs have such choice…

90% of people choose Git because everyone else uses it. Noone has been fired for buying IBM.

I suspect many of us use Git because it was made by Linus Torvalds ( I think everyone agrees that he is a great developer) and is used in the Linux kernel. If it can handle that, then you can be pretty sure it will handle whatever you throw at it (unless you are Facebook, it seems).

Add Github to that mix and you can see why so many developers sleep like babies at night. Of course it's not rigorous, but the choice is not irrational.

Post reply on HN