Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why We Moved Off The Cloud

code.mixpanel.com

Post reply on HN
Print view View original

121–130 of 154 posts

Re: Why We Moved Off The Cloud

#121

Post by asharp »

Joakal wrote: post #70

Interesting Rackspace leaving post: "Rackspace Cloud has had pretty atrocious uptime over the year there has been two major outages where half the internet broke. Everyone has their problems but the main issue is we see really bad node degradation all the time. We’ve had months where a node in our system went down every single week. Fortunately, we’ve always built in the proper redundancy to handle this. We know this…

Amazon.com was only recently transitioned, but AWS was used to host ecommerce apps for partners, iirc. before it's public debeau. This is what it was built for.

Re: Why We Moved Off The Cloud

#122

Post by asharp »

jbyers wrote: post #21

A factor this post doesn't mention is bandwidth cost. If you use a lot of bandwidth and negotiate competitive hardware pricing, you also save with dedicated hosts. Say you have 10 machines at SoftLayer and use 30TB a month. Each machine comes with 3TB and you pool your bandwidth for $25 per server so you can allocate the whole 30TB to your proxies. It's unknown what fraction of your server cost is applied to bandwidt…

Do any American clouds give you 95/5 billing?

Re: Why We Moved Off The Cloud

#123

Post by richardw »

watty wrote: post #108
jread wrote: post #96

Earlier quoted context omitted.

Infrastructure as a service (IaaS) cloud generally implies a few distinguishing features such as hourly utility billing and no long term commitments, rapid elasticity (provision/de-provision in minutes, not hours) and usually (but not always - there are bare metal clouds) virtualization. People often refer to software as a service (SaaS) as cloud, which I think is where it starts getting very ambiguous (i.e. is any h…

I used to fight it but it's a lost cause. The cloud is officially a buzzword and can be used to describe any hosted solution.

Same. Anything that isn't actually solely nailed down to your own hard drive is now 'cloud'. I think we need to find terms that marketers don't get excited by, and thus won't 'embrace and extend' to fit their needs.

Re: Why We Moved Off The Cloud

#124

Post by Loic »

If you are running on top of your dedicated servers, the providers are now all providing a dedicated VLAN for your servers. This allows you to deploy your own VM management software on top of your dedicated servers.

If you can, I recommend you to use Ganeti with Xen or KVM (I use KVM). Rigorous development, very friendly developers and very well designed tools. No wonder it is used internally at Google.

http://code.google.com/p/ganeti/ - Project page.

http://notes.ceondo.com/ganeti/ - Notes on how to use it with Debian (long).

Re: Why We Moved Off The Cloud

#125

Post by asharp »

Interesting.

Also interesting what is simply an artifact from the fact that none of the current "clouds" out there were built to deal with, well, actual loads.

Some things are small, but seem rather strange. Why does no cloud give out 95/5 billing? Why isn't there more resource limiting/etc?

I see a bunch of things leaking out of EC2. People forget that EC2 was designed to deal with large numbers of stateless servers and it's not good for much else. They take the limitations of that and the rest of the AWS platform and apply it to the 'cloud' overall.

Two examples would be from the 'variability' section. CPU limiting under XEN (the hypervisor used by both Amazon and Rackspace) is trivial. The fact that CPU is so variable, especially for smaller tiers, is thus rather interesting.

Similarly with IO. With Rackspace, you are on local disks. As such, unlike Amazon, Rackspace has no defendable reason for being able to starve other users of disk IO.

Also, just as a general data point. There is no real reason why a cloud should be in the same order of magnitude of cost as anything you could touch. Fairly simple reasoning, everything they buy is at massive scale, and there is a very minimal fixedish management cost to deal with all the hardware. What you can work out is that even given almost list prices you are still looking at thousands of percent ROI on cloud servers. What that then says about the market is that there is a current monopoly due to a lack of cloudsmithing knowhow which is the cause of the current situation. Over time, I would expect cloud products to simply dominate standard dedicated servers/colocated servers for most applications.

Re: Why We Moved Off The Cloud

#126

Post by Huppie »

necro wrote: post #114
z2amiller wrote: post #71

Earlier quoted context omitted.

I've always thought the "Drive to the datacentre" argument was BS. If you're writing your app for the cloud, you have to deal with spurious instances going away, degrading, etc. It is no different in the datacenter. If you're driving to the datacenter in the middle of the night to replace a disk or a fan, you're doing it just as wrong as if getting evicted from an EC2 instance causes you to have to scramble oncall re…

I have to agree. I put our stuff in a colo 2 years ago and never looked back. Pretty much all servers come with some kind of remote console interface IPMI, and that's not terminal redirection, thats actually a totally self contained microprocessor and ether port that you can run on a separate subnet and control your server even if it's off. I updated the bios, reinstalled OS's, all via IPMI which is part of the mothe…

I have NEVER had to drive/fly to fix anything. Never even had to use remote hands for anything. Sure some drives died, but standby drives are in place.

Consider yourself lucky. We thought the same thing, but when a RAID controller died on us recently we really didn't know what hit us. It didn't just stop working, it started by hanging the server every now and then, then after a day slowly corrupting drives, then after a day or two it stopped completely.

Re: Why We Moved Off The Cloud

#127

Post by asharp »

papercruncher wrote: post #49
jedberg wrote: post #48

You didn't abandon "the cloud", you just switched providers. You're still paying someone else for servers that you don't own (unless softlayer ships you those machines after 3 years). This is why I hate the term "the cloud" -- because it is too nebulous and non-descriptive.

Cloud implies running in virtualized environment. Dedicated implies that only your bits run on that hardware which is huge for I/O.

Not for most situations.

I work over at orion, and our standard cloud VM's dominate dedicated hardware in IO. There really isn't much competition.

You find most of the 'virtualisation is bad for I/O' is either 'oversold I/O is bad for io' or 'Trying to push all IO for every VM on a box as well as all WAN traffic through one gigE port is bad for io'.

Re: Why We Moved Off The Cloud

#130

Post by asharp »

spydum wrote: post #101
jedberg wrote: post #52

Earlier quoted context omitted.

Also, you don't necessarily take a huge hit in I/O with virtualization. VMWare on dedicated hardware with all the virtualization extensions will be pretty comprable and a lot easier to maintain then straight up raw iron.

To be fair, this is only true if the storage back end is appropriately configured. You will take an I/O hit when instead of a single physical machine asking for a set of sequential blocks off the disks, you have 20 virtual machines asking for seemingly random blocks off the disks. Replace disks with storage array if you'd like.. but the fact remains: more VM's will mean more storage contention. If you have the funds…

Your point about having multiple VM's forming additional seeks is correct, but it misses the bigger picture.

Basically, on any reasonably sophisticated hosting infrastructure those arn't a problem.

If your problem is just seq->random conversion due to additional VMs then bcache/flashcache does a surprisingly good join at making that just plain go away with little additional cost.

On any reasonably sophisticated host you have a DSAN, which has a cool little stats trick, which basically means that as you add additonal vm's together the variance on the total IO load drops, and the load pattern itself becomes more and more normal, the larger and more uncorrelated you get.

That gives each vm more 'burst' capacity when required with many fewer failures (ie. the vm asks for more IO then the current capacity of the system).

This leads to a bunch of interesting stuff when you try to apply it in real world systems, either in HPC or in clouds.

Post reply on HN