Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How do big web sites roll out new versions?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Ask HN: How do big web sites roll out new versions?

#41

Post by unrealwh »

seldo wrote: post #21

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different. Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy t…

haha thats the way it was done at yahoo. go ask HRC now, they'll probably tell you that you are being migrated to a VM-based platform. i'm not speaking derisively, its long time yahoo and other web companies backed off of the ridiculous policies of acquiring servers on a daily basis. the $40mln spend on hardware for panama won't be repeated

Re: Ask HN: How do big web sites roll out new versions?

#42

Post by unrealwh »

sh1mmer wrote: post #32
neilk wrote: post #29

Earlier quoted context omitted.

instead we buy and set up an entirely new set of servers What property do you work on? How is it more efficient to buy or reallocate entirely new servers than update your old ones? How many times per year do you even update your application? The cost of getting the servers ready and installing packages would be prohibitive if you had to roll out, say, an urgent security fix.

I think it depends on the kind of upgrade. Seldo is describing a major migration to a new platform. In the case of a point release a rolling migration or the type I described above are the norm. When doing a major release seldo is spot on. Especially given the small amount of time it takes for hardware to get out of date it's often cheaper to provision new hardware prior to the major release and then deploy straight…

name one yahoo property that bought completely new hardware for a new version rollout

Re: Ask HN: How do big web sites roll out new versions?

#43

Post by unrealwh »

tlrobinson wrote: post #30

Here's a couple of interesting videos, obviously very specific to the particular apps: "Developing Erlang at Yahoo" http://bayfp.blip.tv/file/1281369/ - Talks about the transition of Delicious from Perl to Erlang, including live migrations of the DB. "Taking Large-Scale Applications Offline" http://www.youtube.com/watch?v=cePFlJ8sGj4 - in particular, the section on "Versioning and phased rollouts: what to do when you…

for the last time, delicious does not run on erlang. erlang was used to in part of the migration of data

Re: Ask HN: How do big web sites roll out new versions?

#44

Post by sh1mmer »

neilk wrote: post #39
seldo wrote: post #36

Earlier quoted context omitted.

Exactly right. As I said at the beginning of my post, the "new boxes" method is only used for major platform upgrades, of a magnitude that would happen once a year or less for most properties. For the really huge properties, the cross-colo upgrade pattern you (sh1mmer) describe is the way to go.

Ok, I see what you are saying. Maybe it makes sense if you were switching operating systems or were upgrading PHP from say 4.x to 5.x. I'd still like to see the numbers behind it though. Also, cross-colo is not a factor of being big, it's a factor of having a BCP (business continuity plan). Every team at Yahoo is urged to have BCP for practically everything. I used to do cross-colo upgrades for a relatively small pro…

I think cross-colo is a size thing. BCP is good for all business, but most start ups can't afford to pay for hosting in several data centers.

I image cheap way to fake multiple colos now is to use visualization, amazon style. Heck you could do exactly what Seldo was suggesting every time on EC2.

Re: Ask HN: How do big web sites roll out new versions?

#45

Post by dcurtis »

cheez80 wrote: post #28
dcurtis wrote: post #26

Earlier quoted context omitted.

Wow, that's fascinating. If anyone has any more info on this, I'd really like to learn more.

found it here: http://www.cennydd.co.uk/2008/can-we-avoid-redesign-backlash... "In a nutshell, a meaningless background was removed from a seller page. Pandemonium. After strong resistance the background was reinstated, to everyone’s satisfaction. In fact, the rebellious users were so placated that they failed to notice the designers slowly adjusting the background’s hex values over the next few months. The backgroun…

More info is here, linked from the above article: http://www.uie.com/articles/death_of_relaunch/

Look under the title "eBay.com: Satisfying their Users with Gradual Evolution"

Re: Ask HN: How do big web sites roll out new versions?

#46

Post by seldo »

unrealwh wrote: post #42
sh1mmer wrote: post #32

Earlier quoted context omitted.

I think it depends on the kind of upgrade. Seldo is describing a major migration to a new platform. In the case of a point release a rolling migration or the type I described above are the norm. When doing a major release seldo is spot on. Especially given the small amount of time it takes for hardware to get out of date it's often cheaper to provision new hardware prior to the major release and then deploy straight…

name one yahoo property that bought completely new hardware for a new version rollout

Widgets, Profiles, YDN and Mobile, in the last 2 years. And those are just the ones that I know about because I worked on them. YAP was also new hardware but it was also a new product.

Re: Ask HN: How do big web sites roll out new versions?

#47

Post by [deleted] »

unrealwh wrote: post #42
sh1mmer wrote: post #32

Earlier quoted context omitted.

I think it depends on the kind of upgrade. Seldo is describing a major migration to a new platform. In the case of a point release a rolling migration or the type I described above are the norm. When doing a major release seldo is spot on. Especially given the small amount of time it takes for hardware to get out of date it's often cheaper to provision new hardware prior to the major release and then deploy straight…

name one yahoo property that bought completely new hardware for a new version rollout

[deleted]

Re: Ask HN: How do big web sites roll out new versions?

#48

Post by IsaacSchlueter »

seldo wrote: post #46
unrealwh wrote: post #42

Earlier quoted context omitted.

name one yahoo property that bought completely new hardware for a new version rollout

Widgets, Profiles, YDN and Mobile, in the last 2 years. And those are just the ones that I know about because I worked on them. YAP was also new hardware but it was also a new product.

seldo's right.

Also, the new Buzz site used all new hardware (and was a new product) and Games rolled out on new hardware when it was upgraded from HF2K to PHP. The old servers were reimaged and folded in with their PHP siblings once it was clear that the build was holding up properly.

Re: Ask HN: How do big web sites roll out new versions?

#49

Post by IsaacSchlueter »

unrealwh wrote: post #41
seldo wrote: post #21

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different. Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy t…

haha thats the way it was done at yahoo. go ask HRC now, they'll probably tell you that you are being migrated to a VM-based platform. i'm not speaking derisively, its long time yahoo and other web companies backed off of the ridiculous policies of acquiring servers on a daily basis. the $40mln spend on hardware for panama won't be repeated

What are you talking about? Seldo works at yahoo right now, and has been a part of several product roll-outs.

Re: Ask HN: How do big web sites roll out new versions?

#50

Post by seldo »

dcurtis wrote: post #24
seldo wrote: post #21

For a moderate-sized upgrade, the description given by gaius is pretty accurate. Working at Yahoo, I've been around for a couple pretty huge property changes, and then the deployment process is very different. Basically, hardware is much cheaper than downtime, and very big web companies have lots more money. So we don't swap out old servers gradually: instead we buy and set up an entirely new set of servers, deploy t…

What is the magnitude of this, and how many servers do you have to buy to do the deployment? Also, while the DNS is switching there would be some major database inconsistencies between the two server stacks. How do you deal with that?

The number of servers you have to buy is totally property-dependent. It can be anywhere from 2 to several hundred, but is usually less than 20 in total for the median-sized Yahoo property.

Could you clarify what you mean by database inconsistencies between stacks?

Post reply on HN