Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why We Moved Off The Cloud

code.mixpanel.com

Post reply on HN
Print view View original

21–30 of 154 posts

Re: Why We Moved Off The Cloud

#21

Post by jbyers »

A factor this post doesn't mention is bandwidth cost. If you use a lot of bandwidth and negotiate competitive hardware pricing, you also save with dedicated hosts.

Say you have 10 machines at SoftLayer and use 30TB a month. Each machine comes with 3TB and you pool your bandwidth for $25 per server so you can allocate the whole 30TB to your proxies. It's unknown what fraction of your server cost is applied to bandwidth, but we know the point where you start saving.

At Amazon 30TB of US-East EC2 bandwidth costs 10000 x .12 + 20000 x .09 = $3000.

If you estimate the bandwidth portion of your SoftLayer server cost at less than $275 you're saving money when using your full bandwidth allocation. With servers starting at $159, sub-$100 seems realistic.

In our case with dozens of servers and ~60 TB of bandwidth, we're saving thousands a month compared to EC2.

Re: Why We Moved Off The Cloud

#22

Post by crag »

noodle wrote: post #4
bryanh wrote: post #2

Just about everyone and their mom hopped onto the cloud bandwagon there for good time, but with this steady onslaught of praise for bare-metal hosting, maybe this will reverse a little. It seems like the only thing cloud really provides best is for: 1) Short lived instances or "now" instances. 2) and... what? I'm trying to think of other situations where cloud beats bare metal, but I am coming up short.

> I'm trying to think of other situations where cloud beats bare metal, but I am coming up short. smaller businesses where economies of scale don't kick in, and/or smaller businesses that want to hedge their bets on growth.

Except of course when that small business relays to much on the cloud. I know a local gym that decided to "host" their cash register. I told the owner it was a mistake. He basically said I was suck in the past.

Until his Comcast went/slowed down.

Or a local non-profit who's board came up with a great way to save money - host their phone system in the cloud. A local carrier was happy to sign them to a 3 year contact. Even supplied the 42 phones. Now, they lucky if they can make calls mid day. It's so bad, that if 10 phones are in use, the next call will sound like you calling from a wind tunnel. And forget about calling at peak times. What does the carrier suggest? Upgrading to a T1. Of course that carrier never mentioned this when selling the service in the first place. And personally, with 42 phone plus 50+ computers and other devices I'm suggesting T3 (cost down here about 500 - 600 a month).

My point is, our infrastructure (at least in South Florida) isn't there yet. Sure the cloud is a great idea. But if you can't reach it, it's useless. But that doesn't stop the marketing. Or the complaints.

Re: Why We Moved Off The Cloud

#23

Post by krobertson »

larrys wrote: post #16

"You would think that you could get better prices by signing 1 or 2 year contracts, but interestingly enough, out of the initial 5 providers we talked to the two that didn’t require contracts had the best prices." (snip) "We’ve moved 100% of our machines that rely upon performant (sic) disks to dedicated servers hosted at Softlayer. Roughly speaking, this corresponds to about 80% of our hosting costs. Eventually, we’…

There are other aspects of a contract that can be beneficial:

SLA, including compensation for outages and outs if they have too many. Sure, without a contract you can leave anytime, but a contract isn't necessarily a permanent trap. They are negociable. You can push for lower incidence allowances, an opt-out part way through the contract, and so on.

Support... more in actual human assistance with the move and issues. Depending on your size and the terms, that contract could be worth 6 to 7 figures to the company. That is some serious motivation to help make the initial experiences good and the help along the way.

Re: Why We Moved Off The Cloud

#24

Post by benologist »

jbyers wrote: post #21

A factor this post doesn't mention is bandwidth cost. If you use a lot of bandwidth and negotiate competitive hardware pricing, you also save with dedicated hosts. Say you have 10 machines at SoftLayer and use 30TB a month. Each machine comes with 3TB and you pool your bandwidth for $25 per server so you can allocate the whole 30TB to your proxies. It's unknown what fraction of your server cost is applied to bandwidt…

Yeah bandwidth can be insanely expensive on the cloud. Amazon at least can mitigate that if you're exclusively on their network so it's all incoming + across-their-network but as soon as you start kicking terabytes off their network you're going to feel it.

Dedicateds with massive bandwidth plans are very easy to come by, on top of the perks of dedicated io and raid (which also starts very cheap).

Re: Why We Moved Off The Cloud

#25

Post by ryanlchan »

Do you rent your house, or did you buy it?

Cloud is renting servers: low cap-ex but high op-ex, minimal risk exposure, highly nimble. Dedicated hardware is buying servers: high cap-ex but low op-ex , more risk, and more consistent.

There's nothing inherently "better" in either strategy; they each suit a different need.

Re: Why We Moved Off The Cloud

#26

Post by omfg »

MichaelApproved wrote: post #18

"We recently added a new backup machine with a crappy CPU, little RAM, and 24 2TB drives in a hardware RAID 6 configuration. You can’t get that from a cloud provider and if you find something similar it’s going to cost an order of magnitude more than what we’re paying." I tend to agree with his points but, for back ups the cloud is perfect. If he stayed in the cloud, he wouldn't even need the server in question.

That's assuming the per GB monthly charge works out cheaper than the up front investment in a machine like that. Which these days is rather small.

Re: Why We Moved Off The Cloud

#27

Post by tabizzle »

Can the multi-tenant problem be solved by just using the beefiest EC2 instance available? At some point don't you become the only one occupying that box? And if your site has the volume that MixPanel does, I assume you wouldn't be exposing yourself to single-point-of-failure issues because you'd still have many such boxes. Can someone more knowledgeable please address these?

Re: Why We Moved Off The Cloud

#28

Post by snorkel »

Since their app is highly optimized, profiled, and tweaked low level C then it's no wonder they could not tolerate the CPU variances of the cloud. Even so at least on EC2 there's much less noisy neighbor issues on the bigger CPU instances, for example on extra large instances you essentially have the entire server to yourself.

Re: Why We Moved Off The Cloud

#29

Post by ridruejo »

Thanks for sharing the experience. In your particular case it seems the right decision to move to a dedicated provider. The reasoning was similar to GitHub's (https://github.com/blog/493-github-is-moving-to-rackspace) Cloud environments make the most sense at either end of the spectrum. It works if you are Netflix, where the choice is not to purchase a few servers but build your own Cloud infrastructure (http://perfcap.blogspot.com/2011/08/i-come-to-use-clouds-not...) or if you are getting started, when you are cash-constrained and there are a lot of uncertainties, allows you to run quick experiments etc. Take our case, we provide a cloud hosting tool (http://bitnami.org/cloud) but we don't run all of our systems there, they are divided between "traditional" providers (one of them Softlayer) and AWS. It is a bit of a pain, but for our current requirements and budget, it works nicely.

Re: Why We Moved Off The Cloud

#30

Post by ridruejo »

jbyers wrote: post #21

A factor this post doesn't mention is bandwidth cost. If you use a lot of bandwidth and negotiate competitive hardware pricing, you also save with dedicated hosts. Say you have 10 machines at SoftLayer and use 30TB a month. Each machine comes with 3TB and you pool your bandwidth for $25 per server so you can allocate the whole 30TB to your proxies. It's unknown what fraction of your server cost is applied to bandwidt…

Yes, that is also an issue for us, we provide a large number of downloads (virtual machines, installers) that go into the TB and it would be quite expensive to move that over to EC2. One nice feature of Amazon, if you can afford it, is the ease of use with which you can put that content on their CDN
Post reply on HN