Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why We Moved Off The Cloud

code.mixpanel.com

Post reply on HN
Print view View original

101–110 of 154 posts

Re: Why We Moved Off The Cloud

#101

Post by spydum »

jedberg wrote: post #52
papercruncher wrote: post #49

Earlier quoted context omitted.

Cloud implies running in virtualized environment. Dedicated implies that only your bits run on that hardware which is huge for I/O.

Also, you don't necessarily take a huge hit in I/O with virtualization. VMWare on dedicated hardware with all the virtualization extensions will be pretty comprable and a lot easier to maintain then straight up raw iron.

To be fair, this is only true if the storage back end is appropriately configured.

You will take an I/O hit when instead of a single physical machine asking for a set of sequential blocks off the disks, you have 20 virtual machines asking for seemingly random blocks off the disks.

Replace disks with storage array if you'd like.. but the fact remains: more VM's will mean more storage contention. If you have the funds to have dedicated arrays per VM, hats off to you. Most people never do this, and I/O suffers a penalty. Virtualization has is price, and even that being said, I think it's worth it for most people.

Re: Why We Moved Off The Cloud

#102

Post by herbivore »

benologist wrote: post #24
jbyers wrote: post #21

A factor this post doesn't mention is bandwidth cost. If you use a lot of bandwidth and negotiate competitive hardware pricing, you also save with dedicated hosts. Say you have 10 machines at SoftLayer and use 30TB a month. Each machine comes with 3TB and you pool your bandwidth for $25 per server so you can allocate the whole 30TB to your proxies. It's unknown what fraction of your server cost is applied to bandwidt…

Yeah bandwidth can be insanely expensive on the cloud. Amazon at least can mitigate that if you're exclusively on their network so it's all incoming + across-their-network but as soon as you start kicking terabytes off their network you're going to feel it. Dedicateds with massive bandwidth plans are very easy to come by, on top of the perks of dedicated io and raid (which also starts very cheap).

With many (if not most) reputable dedicated server providers nowadays Inbound traffic is not calculated towards your overall traffic. In other words, it's free and generally unlimited.

Re: Why We Moved Off The Cloud

#103

Post by rhizome »

pablasso wrote: post #14
powertower wrote: post #11

> One last thing about getting dedicated hardware. It's cheaper a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we're not even co-locating. I've been saying this for ages, and every time people would fall over backwards trying to defend/prove their cloud mistake... "The cloud is cheaper, faster, and infinitely scalable." Except none of thos…

I don't think people argue solely on hardware specs, you should count the maintenance and related salaries into the total cost.

Any technical founder or CTO should be able to administer an application box and have some DBA skills.

Re: Why We Moved Off The Cloud

#104

Post by james33 »

kqueue wrote: post #90
AdamGibbins wrote: post #87

Earlier quoted context omitted.

The quality of service and network is also widely different and varied. Hetzner are awesome for the cost, don't get me wrong. But at times their network is terrible and their service seems to vary hugely. You don't get these problems with Softlayer, you pay significantly more and get significantly better almost-guaranteed service.

I see. I haven't tried them personally, my friend has been using them for a year now and so far so good. He did complain about their support though.

We've been using 100TB exclusively for over 2 years now and absolutely no problems. Their support has been top-notch in my opinion.

Re: Why We Moved Off The Cloud

#105

Post by ChuckMcM »

rachelbythebay wrote: post #91
ChuckMcM wrote: post #20

Earlier quoted context omitted.

So I'm running operations at Blekko, and just prior to that I was at Google in their eng/ops organization. I had been doing a whole lot of 'total cost of ownership' aka TCO computations around engineering infrastructure both for Google and of course now for Blekko. The conclusion I came to is that for a 'web 2.0' type setup, the break even point was about 500 'machines.' That was in part because a 'machine' today has…

Beware the "same config everywhere" approach. It works up to a point, then it turns into a disaster. All you need is one totally broken change like "chmod -x /usr" to really make life interesting. You start bleeding machines and pretty soon you have nowhere left to host your tasks. It's interesting, right? At first, you can handle a couple of totally mixed-up machines. Then it stops scaling and you have to start doin…

Yes, I have a white paper, 'Uniformity Considered Harmful' which addressed some of those issues 'in the large'. The interesting bits are two fold, one is where you draw the lines around the 'meta-computer' and how you approach balancing flexibility about provisioning, around utilization in production. I'll see if I can get blog posting up on this stuff.

Re: Why We Moved Off The Cloud

#106

Post by jwegan »

powertower wrote: post #97
jwegan wrote: post #63

Rackspace's primary business is dedicated hosting. In fact they let you have both dedicated hosts and cloud hosts and allow dedicated and cloud hosts to talk to each other. I'm curious why they didn't just switch to Rackspace's dedicated hosting. It would have given them the performance they needed while retaining the flexibility of being able to quickly spin up cloud machines in the same datacenter as the dedicated…

> In fact they let you have both dedicated hosts and cloud hosts and allow dedicated and cloud hosts to talk to each other. That's not amazing at all. You do realise that a "cloud" host and a dedicated box are two exact same hardware boxes siting next to each other in a rack? One's just virtualized 10x with Xen, VMWare, KVN, etc.

Of course its not amazing, but it is a feature they have that would have made the transition to dedicated easier.

Re: Why We Moved Off The Cloud

#107

Post by benologist »

herbivore wrote: post #102
benologist wrote: post #24

Earlier quoted context omitted.

Yeah bandwidth can be insanely expensive on the cloud. Amazon at least can mitigate that if you're exclusively on their network so it's all incoming + across-their-network but as soon as you start kicking terabytes off their network you're going to feel it. Dedicateds with massive bandwidth plans are very easy to come by, on top of the perks of dedicated io and raid (which also starts very cheap).

With many (if not most) reputable dedicated server providers nowadays Inbound traffic is not calculated towards your overall traffic. In other words, it's free and generally unlimited.

I did not know that. My provider counts inbound which sucks because that's most of our traffic but we run at about 6 out of 10 terabytes per server per month so it's not a problem... for now.

Re: Why We Moved Off The Cloud

#108

Post by watty »

jread wrote: post #96
jedberg wrote: post #48

You didn't abandon "the cloud", you just switched providers. You're still paying someone else for servers that you don't own (unless softlayer ships you those machines after 3 years). This is why I hate the term "the cloud" -- because it is too nebulous and non-descriptive.

Infrastructure as a service (IaaS) cloud generally implies a few distinguishing features such as hourly utility billing and no long term commitments, rapid elasticity (provision/de-provision in minutes, not hours) and usually (but not always - there are bare metal clouds) virtualization. People often refer to software as a service (SaaS) as cloud, which I think is where it starts getting very ambiguous (i.e. is any h…

I used to fight it but it's a lost cause. The cloud is officially a buzzword and can be used to describe any hosted solution.

Re: Why We Moved Off The Cloud

#109

Post by mnutt »

imbriaco wrote: post #84
towelrod wrote: post #61

Earlier quoted context omitted.

I think the OP just meant that once someone like reddit moves into your cloud, your performance goes down. Meaning, the hardware that my little blog is on works great until that hardware also hosting reddit, and then my blog takes a beating because reddit is thrashing away with actual activity.

Not if the cloud you're running your workload on is large enough. The Amazon Web Services cloud could absorb many Reddit sized sites without any other customers experiencing an impact. AWS is _enormous_.

Not in the sense of running out of machines, but at least from my past experiences if your VMs share physical hardware with someone who is using their maximum IO, yours will suffer as well.

It makes it more challenging to load test. When there is no contention you can usually 'burst' to use more of the machine's resources, but you can't necessarily trust that you will always have that capacity.

Post reply on HN