Ask HN: How do big web sites roll out new versions?

I've done rolling deployments but also a similar but different way at Yahoo!.

We have access to multiple colos of slightly over-capacity machines. It's not uncommon to migrate by switching all traffic over to one colo. Then upgrade the next colo to the new code and switch all traffic there. Finally upgrade the first colo and then start cross loading the traffic between colos again.

Since we are big enough to have at least a couple of colos per geographic region this means we can avoid rolling migrations which are pretty horrible when you get trapped in a failure mid-roll.

Well, when Google App Engine comes into play, this problem isn't big. They have something called versioning (in the admin panel) which allows you to change the app to the next version or revert back to an old version.

From an amateur POV: But when usual servers come into play, I have had no clue abt big ones.

Infact a couple of months back, when my home server had a burst of traffic from my mobile app, it couldn't handle it (256kbps connection, 256MB ram with a 1.3 GHz P3 processor. well suited to serve a high bandwidth mobile app of 1000 users and also for constantly collecting data from APIs). But due to sudden traffic outburst, I had to string a friend's comp to mine.

My procedure: I first stopped updating data from APIs to my DB and kept it static. No updating the DB. Copied it to my friend's fast comp. I had my computer interfaced with a mobile which accepts input from the user mobile phones. That was the app's requirement. So I had an advantage of lining up requests right there in the mobile phone while I was switching the server. That trick should help if you are developing a mobile app on a very limited resource (a hobby project).

Here is basically what one of our big financial clients [1] does twice per year:

Integration, or sytem testing had taken place over the previous 1-2 month period.

They have 2 large datacenters, let's call one "production" and the other one "disaster recovery" (abbreviated "DR"). Between these and the internet are some large routers (the Cisco ones that cost about what a house costs). At the beginning of "migration" the routers are switched to point from production to DR, so that all internet traffic points to DR.

At this time, the servers are being updated with the new code. This can take some time, especially if large databases need to be restored as part of the migration, or if the update scripts take a long time. There are a lot of servers involved, a ballpark is about 100 servers: some Sun, some WinTel, some IBM mainframes. Some mirrored, some clustered, some all by their little old lonesome. Some applications are Java, some .NET, and some Cobol[2].

Approximate timeline [3]:

People start dialing into the main conference call about 5:30PM eastern [4].

Switch to DR about 6pm eastern.

Code in production is migrated/installed, servers rebooted if necessary. Done about 9-11pm eastern.

Testing [5] starts and continues until about 3AM.

GO/NO-GO decision is made sometime between 3 and 4AM[6].

Rollback if necessary.

Switch routers to point to production at 6AM.

Preliminary postmortem report generally done by 2PM.

If no rollback, then repeat the following evening for DR.

Notes: 1 - They're a Fortune 100 company, I'm not telling who they are.

2 - That I'm aware of. It would not surprise me at all that there are a number of other "brands" of servers or programming languages involved.

3 - The actual timeline is usually a spreadsheet that's at a minimum, 50 pages long, plus about 10 more pages of first, second and third contacts in case something bursts into flames.

4 - It is common to have 100+ people monitoring the main conference line, and 1-2 dozen other conference lines used for individual components/products. One has to be awake and alert in case you're "called out" on the conference line.

5 - In general, because the main URL/URI/Hostnames are now pointing to DR, hosts files are changed so that configuration files don't get edited.

6 - Sometimes the decision gets delayed until almost 6AM if there are some problems.

Actually, Amazon is kind of sharding their database: They encapsulate every kind of data into a program that manages it:

>What I mean by that is that within Amazon, all small pieces of business functionality are run as a separate service. For example, this availability of a particular item is a service that is a piece of software running somewhere that encapsulates data, that manages that data.

(see http://queue.acm.org/detail.cfm?id=1388773)

Ok, I see what you are saying. Maybe it makes sense if you were switching operating systems or were upgrading PHP from say 4.x to 5.x. I'd still like to see the numbers behind it though.

Also, cross-colo is not a factor of being big, it's a factor of having a BCP (business continuity plan). Every team at Yahoo is urged to have BCP for practically everything. I used to do cross-colo upgrades for a relatively small project once we had more than two frontend servers.