Ask HN: How do big web sites roll out new versions?

The operations of big websites fascinate me. Sites such as http://highscalability.com/ give an glimpse into the architecture of Amazon, Twitter, and other large systems, but leave out the operations side. So, can anyone explain how they deploy new versions of their sites? For instance, I can't imagine Amazon choosing to go down for even a minute, so what do they do? Am I wrong? Thanks!

Haven't done anything of the like, but perhaps using something like apache mod_proxy to redirect visitors to machines that are already updated?

They'll take a few webservers and their associated middle tier boxes out of service on their front-end loadbalancers, wait for all the sessions to migrate/fail over to others, upgrade them and put them back into service. The loadbalancers will be smart enough to do affinity (put this customer onto this pool of servers if possible). At any one time after the roll-out begins, x% of the customer base will be on the new code where x->100 if things go well, or if something unexpected happens, (100-x)% of customers will never even see it, and the rest (x In apps like this the policy is to only add columns to tables and new tables, never to remove anything, so the database can be upgraded hot, and the old code can continue to run on it, only the new code will see any new columns/tables.

It's totally different for different types of sites. Facebook is different than Amazon is different than Yahoo is different than Flickr.

In general, I'd say, most sites data is sharded or clustered. Amazon, I'm guessing, basically has many many different instances of their app running on different clusters all over the world (multiple clusters per datacenter). So they upgrade a cluster at a time, and their databases all synchronize with others of the same version.

Facebook's data is, obviously, sharded by network. So any given cluster runs x number of networks. Upgrades are then, again i'm guessing here, rolled out network by network. The data layer can be different from network to network.

Most of the time though, updates to large scale services aren't changing db schemas or making huge changes to the data layer, so it's as simple as updating the code and rebooting some app servers.

(Obviously I don't know exactly how they do it, cause I don't work at any of these places, but deploying it's that hard of a problem to solve)