Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do big web sites roll out new versions?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 52 posts

Re: Ask HN: How do big web sites roll out new versions?

#21

Post by seldo »

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different.

Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy to them several weeks in advance of the planned launch, and run QA against these production-level boxes. Then, when we are ready to "launch", all we're really doing is a relatively low-risk DNS change: all the potentially tricky deployment issues having been ironed out beforehand.

After a couple of weeks/months of operation on the new hardware, if there have been no major problems, the old boxes are decommissioned -- either re-imaged and put back into service to expand capacity, or more often taken out of service entirely (I've no idea what we do with old boxes when we stop using them, funnily enough).

Re: Ask HN: How do big web sites roll out new versions?

#22

Post by gaius »

wesley wrote: post #3

Haven't done anything of the like, but perhaps using something like apache mod_proxy to redirect visitors to machines that are already updated?

Right idea, wrong technology, this is done in hardware.

The "secret sauce" is an invisible layer in front of the webservers. The client's connection terminates in this layer, and the loadbalancers establish new sessions with the web servers - the client never actually touches the server. This extra layer lets you do all sorts of clever stuff that was unimaginable back in the day when all we had was Squid and round-robin DNS.

Re: Ask HN: How do big web sites roll out new versions?

#23

Post by bjclark »

gaius wrote: post #19
bjclark wrote: post #13

Earlier quoted context omitted.

How is this contrary to what I said? I didn't say Amazon sharded. What/why would they shard? Clustering != sharding. They most definitely have multiple instances of the app running around the world, which synchronize data with each other, which is why you've never been unable to buy something. It's highly unlikely that all their instances would be down or overloaded at the same time. Also, they most certainly don't h…

The product data will be in multiple physical replicated shared-nothing databases each of which has the entire dataset - a single logical database. The principle of sharding is that each database has a subset of the data and you place some logic in front of it to direct the query to the right place. Now if I'd been able to buy kitchenware but not garden tools one day, then I might say their product database was shard…

Are you still arguing that I said Amazon sharded their database? Because I've re-read what I said, and what you said like 4 times and I can't see where I said they shard.

Re: Ask HN: How do big web sites roll out new versions?

#24

Post by dcurtis »

seldo wrote: post #21

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different. Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy t…

What is the magnitude of this, and how many servers do you have to buy to do the deployment?

Also, while the DNS is switching there would be some major database inconsistencies between the two server stacks. How do you deal with that?

Re: Ask HN: How do big web sites roll out new versions?

#25

Post by jjs »

gaius wrote: post #5

They'll take a few webservers and their associated middle tier boxes out of service on their front-end loadbalancers, wait for all the sessions to migrate/fail over to others, upgrade them and put them back into service. The loadbalancers will be smart enough to do affinity (put this customer onto this pool of servers if possible). At any one time after the roll-out begins, x% of the customer base will be on the new…

If the upgrade is expected to change the user experience, this is also a great time to do some A/B testing across versions.

Re: Ask HN: How do big web sites roll out new versions?

#26

Post by dcurtis »

cheez80 wrote: post #4

i remember reading somewhere that ebay changed the background color of their homepage from gray to white, gradually, over 6 months, as not to alienate customers who were used to the gray. can't provide links, and sorry because this isn't really related. i just thought that was interesting. :)

Wow, that's fascinating. If anyone has any more info on this, I'd really like to learn more.

Re: Ask HN: How do big web sites roll out new versions?

#27

Post by [deleted] »

gaius wrote: post #5

They'll take a few webservers and their associated middle tier boxes out of service on their front-end loadbalancers, wait for all the sessions to migrate/fail over to others, upgrade them and put them back into service. The loadbalancers will be smart enough to do affinity (put this customer onto this pool of servers if possible). At any one time after the roll-out begins, x% of the customer base will be on the new…

[deleted]

Re: Ask HN: How do big web sites roll out new versions?

#28

Post by cheez80 »

dcurtis wrote: post #26
cheez80 wrote: post #4

i remember reading somewhere that ebay changed the background color of their homepage from gray to white, gradually, over 6 months, as not to alienate customers who were used to the gray. can't provide links, and sorry because this isn't really related. i just thought that was interesting. :)

Wow, that's fascinating. If anyone has any more info on this, I'd really like to learn more.

found it here:

http://www.cennydd.co.uk/2008/can-we-avoid-redesign-backlash...

"In a nutshell, a meaningless background was removed from a seller page. Pandemonium. After strong resistance the background was reinstated, to everyone’s satisfaction. In fact, the rebellious users were so placated that they failed to notice the designers slowly adjusting the background’s hex values over the next few months. The background got lighter and lighter until one day—pop!—it was gone."

Re: Ask HN: How do big web sites roll out new versions?

#29

Post by neilk »

seldo wrote: post #21

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different. Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy t…

instead we buy and set up an entirely new set of servers

What property do you work on? How is it more efficient to buy or reallocate entirely new servers than update your old ones?

How many times per year do you even update your application? The cost of getting the servers ready and installing packages would be prohibitive if you had to roll out, say, an urgent security fix.

Re: Ask HN: How do big web sites roll out new versions?

#30

Post by tlrobinson »

Here's a couple of interesting videos, obviously very specific to the particular apps:

"Developing Erlang at Yahoo" http://bayfp.blip.tv/file/1281369/ - Talks about the transition of Delicious from Perl to Erlang, including live migrations of the DB.

"Taking Large-Scale Applications Offline" http://www.youtube.com/watch?v=cePFlJ8sGj4 - in particular, the section on "Versioning and phased rollouts: what to do when you can't assume all servers have the same code"

Post reply on HN