Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Mercurial at Facebook

code.facebook.com

Post reply on HN
Print view View original

211–220 of 245 posts

Re: Scaling Mercurial at Facebook

#211

Post by thrownaway2424 »

Touche wrote: post #203
thrownaway2424 wrote: post #159

Earlier quoted context omitted.

Immediate regressions are good! If someone at Google breaks my code, I will know within half an hour at the latest and I will tell them to go fix it or just revert their changes myself. Immediate regressions also go perfectly with daily (or hourly!) releases. If there's a performance problem it will be identified early and I will only have thousands of changes to investigate instead of tens of millions. Imagine if I…

I don't just mean performance regressions. Someone upstream can change an API in a way that doesn't fit well with your use-case, goes in and "fixes" your code (makes sure all the tests pass) to fit the new API but makes it less maintainable in the process.

Teams (at Google) can't change calling code without the review and approval of the owners of the calling code, so what you state would not happen.

Re: Scaling Mercurial at Facebook

#212

Post by agwa »

novaleaf wrote: post #194
randartie wrote: post #156

Earlier quoted context omitted.

The discussion appears deleted on that thread

fyi, it was working about 8hrs ago. broken now. guess someone didn't like the inuendo ;)

It's working at the moment. I do not believe gmane would have deleted the contents of a public mailing list archive like this, so there must have been a transient technical problem.

Re: Scaling Mercurial at Facebook

#213

Post by yonran »

shoover wrote: post #209
yonran wrote: post #41

Earlier quoted context omitted.

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

Nice list. What then are the mechanics of making a breaking change in a library, file format, network protocol, etc. in the single repo system?

Ideally, to make a breaking API change, you change the function and all the references within a single commit. Since this repository is only used for statically compiled programs, there is no need to keep the old API anymore.

For protocols and file formats, Google universally uses protocol buffers with many optional fields. The protocol buffer library’s default is that when you read a protocol buffer, modify it, and write it back out, the fields that you didn’t understand are passed through. This means that middleman servers don’t need to be recompiled when you add new optional fields that they don’t use.

But for the actual client and server, you generally don’t have the luxury of replacing them both at the same time. So you have to add the new field that is disabled using a flag, wait for it to rolled out to both the client and server, then enable the new field and disable the old field using the flag, then remove the flag and old field. It’s something that you coordinate with the release engineers. But it’s not formalized in the software version numbers.

Re: Scaling Mercurial at Facebook

#214

Post by 16bytes »

ahomescu1 wrote: post #181
tonfa wrote: post #169

Earlier quoted context omitted.

I think you are confused, git and hg branches are very similar (I'd try to help but I'm not sure what the confusion is).

It's simple: I created a local branch, worked on it, then tried the git workflow: 1. Switch to the default branch 2. Cherry-pick (with the equivalent hg command) my changes from my own branch into default 3. Push the changes to the remote repo What happened was that my local branch got pushed to the server, along with the default one. With git this wouldn't happen, it would push the local master to the remote master.

This is because hg pushes all changesets by default. There are two workflows in hg that would get you what you want:

hg push -b default

Pushes only changesets from the default branch. You can also do these with phases by marking your branch as private.

Re: Scaling Mercurial at Facebook

#215

Post by kanzure »

khuey wrote: post #133
dubcanada wrote: post #120

Earlier quoted context omitted.

I think they just want to modify Git and don't have any solid C developers that can make such things. So they turned to a python solution which is perfectly fine. But webkit, and chromium, as well as other GIANT projects which as far as I know are larger then Facebook seem to work fine on Git.

You really believe that Facebook couldn't find a competent C programmer to make some changes to git?

Would you accept an answer that complains about how hard it is to hire programmers? :)

Re: Scaling Mercurial at Facebook

#217

Post by estebank »

exDM69 wrote: post #200
yonran wrote: post #41

Earlier quoted context omitted.

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using…

> * You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using. Git submodules may have a number of problems of its own, but it solves this one. There's always an unambiguous version number, which is the commit hash of the top repo. Every subrepository's commit hash is stored in the top repo an…

In 1.8.2 you can track a branch name instead of a commit hash. This has the benefit of allowing you to work always against the latest version, while foregoing the advantage of having a single unambiguous version number.

Re: Scaling Mercurial at Facebook

#218

Post by com2kid »

ed_blackburn wrote: post #37

I wonder what they use at Microsoft. For their sake I hope they don't subject their own engineers to TFS.

We eat our own dog food, most teams have moved over to TFS by now. I don't know about the larger orgs (Windows, Office) but for smaller groups TFS is how it is.

MS doesn't have a unified build environment, every team generally does their own thing. It has its pluses and minuses.

Shared code would occasionally be useful at Microsoft, but not as often as you'd think. Generally when relying on another team's code, it is preferred to take it as a binary drop when they do a product release, just like any other customer. This helps prevents needing to deal with churn in one's dependencies.

Re: Scaling Mercurial at Facebook

#219

Post by Crito »

mseebach wrote: post #198
r0muald wrote: post #196

Earlier quoted context omitted.

Like building the entire system in PHP, then throwing HipHop at it? Seems to work though.

I'm not convinced that HipHop is an example of that. As I understand it, HipHop allowed Facebook to increase performance hugely with a very modest investment, and very low risk. Rewriting all the critical bits of Facebook in a different language to realize the same speed-up would likely have required a lot more resources and been orders of magnitude more risky.

The difficulty in "rewriting [critical, or otherwise] bits of Facebook" is probably related to the same decisions that make "single repo for everything" the logical choice for the site.

Re: Scaling Mercurial at Facebook

#220

Post by Crito »

moron4hire wrote: post #208
Crito wrote: post #56

Earlier quoted context omitted.

That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go. The problem is "how do you grow even further ?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos. Thankfully very few companies need to worry about this prob…

At what point do we say "enough" though? I mean, start from the absolute maximum: how would one make a system to control all of the source code in the world? For all of the source code in the world that is source-controlled, it exists in separate repos. It doesn't exist in one repo, and where there are inter-project dependencies, it is the dependency consumer's responsibility to keep abreast of the changes in the dep…

This is a very valid point. It is not clear at all that a single repo for the world is desirable in the long run, despite some of the individual advantages that sort of setup has. It is part of the reason that I don't think we will see the "next generation" of VCS anytime soon. (Which in turn makes me think that "holding out" for those systems, hoping they will rescue you from your scaling problem, is a bad decision. It is best to move to multiple repos sooner rather than later).
Post reply on HN