Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why We Moved Off The Cloud

code.mixpanel.com

Post reply on HN
Print view View original

131–140 of 154 posts

Re: Why We Moved Off The Cloud

#131

Post by asharp »

herbivore wrote: post #102
benologist wrote: post #24

Earlier quoted context omitted.

Yeah bandwidth can be insanely expensive on the cloud. Amazon at least can mitigate that if you're exclusively on their network so it's all incoming + across-their-network but as soon as you start kicking terabytes off their network you're going to feel it. Dedicateds with massive bandwidth plans are very easy to come by, on top of the perks of dedicated io and raid (which also starts very cheap).

With many (if not most) reputable dedicated server providers nowadays Inbound traffic is not calculated towards your overall traffic. In other words, it's free and generally unlimited.

Remember that most loads are 1:10 inbound:outbound, so unless you have a strange sort of workload it really doesn't help you much.

If you do happen to have a lot of inbound traffic though, it's a nice little arrangement as the bandwidth may be almost free to the provider (Symetricity requirements) and still very useful to you.

Re: Why We Moved Off The Cloud

#132

Post by asharp »

noelwelsh wrote: post #10
bryanh wrote: post #2

Just about everyone and their mom hopped onto the cloud bandwagon there for good time, but with this steady onslaught of praise for bare-metal hosting, maybe this will reverse a little. It seems like the only thing cloud really provides best is for: 1) Short lived instances or "now" instances. 2) and... what? I'm trying to think of other situations where cloud beats bare metal, but I am coming up short.

Like the blog says, CPU is fairly cheap on the cloud. Memory is fairly expensive and normally good disk IO cannot be purchased (Bytemark's BigV looks different in this regard). We have a service ( http://www.mynaweb.com/ ) that has a bunch of mostly CPU bound services and a few IO bound ones. We also want the CPU bound services to be near to customers. It makes sense for us to acquire cheap cloud instances around the…

Just a question.

Who in the US provides cloud with disks comparable to dedicated servers? (or otherwise 'good')?

I say this because I run a cloud company called orion in Australia, we produce cloud VMs with faster then dedicated disk performance. When we were last in SV pitching, nobody knew of anybody in a similar sort of space.

I'm just asking because you seemed to emphasise 'normally' and 'good'.

Re: Why We Moved Off The Cloud

#133

Post by asharp »

powertower wrote: post #11

> One last thing about getting dedicated hardware. It's cheaper a lot cheaper. We have machines that give us 2-4x performance that cost less than half as much as their cloud equivalents and we're not even co-locating. I've been saying this for ages, and every time people would fall over backwards trying to defend/prove their cloud mistake... "The cloud is cheaper, faster, and infinitely scalable." Except none of thos…

Very few workloads need to scale out past the limits of any reasonable cloud provider.

Properly built clouds can give you IO faster then a dedicated server, and give you CPUS on demand that far exceed what you would otherwise afford (due to you only needing it occasionally).

And any reasonable cloud provider can discount things such that is unreasonable for you to move off. They have a much lower bottom line then a comparable dedicated server provider.

Is interesting that this doesn't seem to be the case currently.

Re: Why We Moved Off The Cloud

#134

Post by asharp »

rachelbythebay wrote: post #91
ChuckMcM wrote: post #20

Earlier quoted context omitted.

So I'm running operations at Blekko, and just prior to that I was at Google in their eng/ops organization. I had been doing a whole lot of 'total cost of ownership' aka TCO computations around engineering infrastructure both for Google and of course now for Blekko. The conclusion I came to is that for a 'web 2.0' type setup, the break even point was about 500 'machines.' That was in part because a 'machine' today has…

Beware the "same config everywhere" approach. It works up to a point, then it turns into a disaster. All you need is one totally broken change like "chmod -x /usr" to really make life interesting. You start bleeding machines and pretty soon you have nowhere left to host your tasks. It's interesting, right? At first, you can handle a couple of totally mixed-up machines. Then it stops scaling and you have to start doin…

You just need to do rolling deployments to make simple things like that go away.

Roll out a deployment to N machines (like say 10), run self checks (you have those, right?), if everything passes give them standard load. Over the next K period of time, periodically check up on them. After that, roll over to N*2 or N^2 nodes, continue until you have rolled out to your entire cluster.

Re: Why We Moved Off The Cloud

#135

Post by adgar »

Loic wrote: post #124

If you are running on top of your dedicated servers, the providers are now all providing a dedicated VLAN for your servers. This allows you to deploy your own VM management software on top of your dedicated servers. If you can, I recommend you to use Ganeti with Xen or KVM (I use KVM). Rigorous development, very friendly developers and very well designed tools. No wonder it is used internally at Google. http://code.g…

> No wonder it is used internally at Google.

Well, I think that's more because it was written at Google.

Re: Why We Moved Off The Cloud

#136

Post by hopeless »

kqueue wrote: post #72

This service provider has great pricing compared to softlayer http://www.hetzner.de/en/hosting/produktmatrix/rootserver-pr... Intel Corei7-2600 Quad core + 16GB DDR3 + 2 x 3TB 7200 for 49 euro.

Wow, that seems like an amazing price. I wonder how bad the network really is? Surely you can't go wrong for a small startup / hobby site at €49/mo

Re: Why We Moved Off The Cloud

#137

Post by Loic »

adgar wrote: post #135
Loic wrote: post #124

If you are running on top of your dedicated servers, the providers are now all providing a dedicated VLAN for your servers. This allows you to deploy your own VM management software on top of your dedicated servers. If you can, I recommend you to use Ganeti with Xen or KVM (I use KVM). Rigorous development, very friendly developers and very well designed tools. No wonder it is used internally at Google. http://code.g…

> No wonder it is used internally at Google. Well, I think that's more because it was written at Google.

Yes, but, it would be a dead horse, they would have not been keeping it in production and improving it over the past 5 years. I think this is the key, well designed, well maintained, well used for critical stuff in a big company all that over several years.

Edit: forgot part of the sentence, stupid me.

Re: Why We Moved Off The Cloud

#138

Post by eli »

Huppie wrote: post #126
necro wrote: post #114

Earlier quoted context omitted.

I have to agree. I put our stuff in a colo 2 years ago and never looked back. Pretty much all servers come with some kind of remote console interface IPMI, and that's not terminal redirection, thats actually a totally self contained microprocessor and ether port that you can run on a separate subnet and control your server even if it's off. I updated the bios, reinstalled OS's, all via IPMI which is part of the mothe…

I have NEVER had to drive/fly to fix anything. Never even had to use remote hands for anything. Sure some drives died, but standby drives are in place. Consider yourself lucky. We thought the same thing, but when a RAID controller died on us recently we really didn't know what hit us. It didn't just stop working, it started by hanging the server every now and then, then after a day slowly corrupting drives, then afte…

This is totally anecdotal, but I've personally had far more problems with bad RAID controllers than with dying hard drives.

Re: Why We Moved Off The Cloud

#139

Post by dhimes »

benologist wrote: post #107
herbivore wrote: post #102

Earlier quoted context omitted.

With many (if not most) reputable dedicated server providers nowadays Inbound traffic is not calculated towards your overall traffic. In other words, it's free and generally unlimited.

I did not know that. My provider counts inbound which sucks because that's most of our traffic but we run at about 6 out of 10 terabytes per server per month so it's not a problem... for now.

It double-sucks because it's not easy (AFAIK) to gzip traffic from web app to the server.

Re: Why We Moved Off The Cloud

#140

Post by cr4zy »

proofpeer wrote: post #15

I think you have to distinguish between where you rent whole server capabilities, and more managed models like Google App Engine. It seems to me that GAE should deliver more consistent performance.

GAE definitely does not deliver more consistent performance. I've done a ton of performance tweaking on GAE as of late and my bottlenecks are now reduced to random points in the code between RPC's where my Python thread is obviously locked out of the CPU it was running on.

I should say that the variation this leads to is at max around two seconds. I believe this is due to App Engine doing some dynamic grouping of slow applications. So if your app has fast response times, it will be grouped with other apps having fast response times, so the maximum downside is limited.

Post reply on HN