How Facebook pushes updates to the site

#1

Post by creativityhurts » Sat, May 28, 2011, 7:05 AM UTC

How Facebook pushes updates to the site
facebook.com

Re: How Facebook pushes updates to the site

#2

Post by baby » Sat, May 28, 2011, 9:44 AM UTC

So facebook is programed in PHP but everything on the server is in C++ thanks to "hiphop"? mind=blown

Re: How Facebook pushes updates to the site

#3

Post by creativityhurts » Sat, May 28, 2011, 9:49 AM UTC

baby wrote: post #2

So facebook is programed in PHP but everything on the server is in C++ thanks to "hiphop"? mind=blown

Yeah, C++ and Java http://www.quora.com/What-programming-languages-are-used-at-...

Re: How Facebook pushes updates to the site

#4

Post by ignifero » Sat, May 28, 2011, 10:13 AM UTC

So they don't really test the code, they just push it out, and fix the bugs on Thursdays. Genius strategy. No wonder their platform breaks so often.

Re: How Facebook pushes updates to the site

#5

Post by jpulgarin » Sat, May 28, 2011, 10:24 AM UTC

ignifero wrote: post #4

So they don't really test the code, they just push it out, and fix the bugs on Thursdays. Genius strategy. No wonder their platform breaks so often.

The code is tested. There are unit tests, Watir tests, tests written by the developer for that specific change, and all changes require a test plan from the engineer.

Re: How Facebook pushes updates to the site

#6

Post by akent » Sat, May 28, 2011, 11:02 AM UTC

I'm pretty impressed by the "push karma" system for gauging how risky individual engineers' commits are on average.

Here's another (much shorter!) video where Chuck Rossi talks about push karma very briefly: https://www.facebook.com/video/video.php?v=778890205865&...

Re: How Facebook pushes updates to the site

#7

Post by shin_lao » Sat, May 28, 2011, 11:15 AM UTC

ignifero wrote: post #4

So they don't really test the code, they just push it out, and fix the bugs on Thursdays. Genius strategy. No wonder their platform breaks so often.

They have tests, but for systems of that complexity, some bugs only appear in production whatever you do.

Re: How Facebook pushes updates to the site

#8

Post by shin_lao » Sat, May 28, 2011, 11:16 AM UTC

What he says at the beginning is to me the most important. Having great tool is great (!), but the most important is to have the right culture about QA and releases.

Re: How Facebook pushes updates to the site

#9

Post by avar » Sat, May 28, 2011, 1:06 PM UTC

It's interesting that their entire release architecture seems to be focused on never pushing bad things out to production, whereas given their traffic they could probably push things out much sooner (minutes after they're committed) to small parts of their overall traffic, and slowly increase the traffic on those pieces of code as they prove themselves to be stable, or quickly revert them if they're not.

That would mean having a lot of versions of Facebook live at any one point, but as those parts prove themselves stable they'd gradually be rolled out to all of their traffic.

One point that also wasn't covered is that as they're pushing things out they'll only cherry-pick parts of their codebase depending on which engineers they have around. I wonder if they have a lot of hairy merge conflicts around release time due to that, and bugs in production resulting purely from those merge conflicts. Or worse, subtle bugs resulting in change A going out, but being programmed against a function that was changed in change B, which is not going out because the author of change B isn't in today.

Re: How Facebook pushes updates to the site

#10

Post by madamepsychosis » Sat, May 28, 2011, 1:15 PM UTC

I like how their entire development cycle revolves around people getting drunk on weekends.