Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

thehftguy.com

Post reply on HN
Print view View original

151–160 of 179 posts

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#151

Post by user5994461 »

doh wrote: post #145
doh wrote: post #117

Earlier quoted context omitted.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

@user5994461 can't reply to your comment for some reason, but you're correct. Ansible would never scale. We ditched it a long time ago. We're using packer to create images/docker and then distribute them automatically via kubernetes or just directly through the custom images. As I've mentioned earlier, we're running an immutable infrastructure. That means once we need to change something we replace the whole server.…

The "reply" button sometimes goes away when the discussion is deep enough. Gotta click the comment to comment.

I would imagine that rolling a container to thousands hosts may take a while.

What kind of load and software do you run? In my experience, dramatically scaling out increases load and latency variance and cause all kind of problems.

When was your experience with SoftLayer? Care to elaborate? I've got some interests in them for future projects. I'd rather hear about the issues now :D

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#152

Post by user5994461 »

late2part wrote: post #141
doh wrote: post #117

Earlier quoted context omitted.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minde…

> I think you're being close minded to building out a 20-60 cabinet data-center to offload some of your workload.

I'd say it's time to forget about the infrastructure. Should optimize the software and rewrite in C++ :D

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#153

Post by user5994461 »

geggam wrote: post #132
doh wrote: post #128

Earlier quoted context omitted.

What kind of servers run at that price range? I've listed [0] some of the reasons why are we hosting in cloud instead of our own datacenter. [0] https://news.ycombinator.com/item?id=13260805

Various servers, i2 for Cassandra eats most of it. Application at various levels, multi region etc etc We too are hockey sticking but not ready to go to metal .... yet. Your reasons make sense. I would grow that team soon :)

To quote another articles from that site: https://thehftguy.com/2016/11/18/google-cloud-is-50-cheaper-...

The i2 comparisons are under "local SSD and scaling up". Google have local 400 GB SSD that can be attached to any instance. That's a lot more flexible and a hell lot cheaper when you have specific needs.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#154

Post by mdekkers »

paulddraper wrote: post #124
doh wrote: post #23

Earlier quoted context omitted.

We couldn't even get them to approve 5k servers on AWS. We run everything in GCE.

That is hard to believe. Their case studies have organizations using thousands or tens of thousands of servers for scientific computing, etc.

Case studies tend to be written by the marketing team, and will tell you what the marketing team wants you to believe. I'll take "hey, this is my hands-on experience" over "this is what we want you to think" everytime.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#155

Post by hruan »

AlexCoventry wrote: post #136
hruan wrote: post #57

Earlier quoted context omitted.

What about N-series VMs on Azure? https://azure.microsoft.com/en-us/blog/azure-n-series-genera...

Last I checked I only found per-month pricing. Are there finer-grained options?

VMs on Azure are billed per minute and I haven't found any indication of N-series being the exception.

What's your source on per-month pricing for N-series?

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#156

Post by banachtarski »

doh wrote: post #117
banachtarski wrote: post #60

Earlier quoted context omitted.

You should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

I'll be honest, it really feels like you haven't done a good job of scaling up first (at least based on your comments about cost and how they aren't provisioning fast enough). When you say you tried OVH and it didn't work, are you deploying dual processor hexacores with 64 gb+ of RAM? Because if not you're probably not doing it right. What software stack are you running?

Get 1-2 more people and move off the cloud. Maybe not for everything but to service your base traffic. You can manage it if you spend a few days learning how, in the same way you learned AWS/GCE.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#157

Post by banachtarski »

phonon wrote: post #122
banachtarski wrote: post #109

Earlier quoted context omitted.

Just the sheer number of instances the OP is using has put the usage FAR into the territory where the premiums spent on virtualized instance costs far outweigh any clever strategy one might use to make things cheaper in the cloud. It has nothing to do with what the infrastructure is, and everything to do with raw volume.

Spot instances are super cheap...about 80% less than regular instances.

Yea but it's still virtualized slow molasses at the end. And still more expensive. My point is that at 25k servers, even a 5% efficiency gain is significant, and I'm willing to bet the margins are still higher.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#158

Post by doh »

late2part wrote: post #149
doh wrote: post #144

Earlier quoted context omitted.

You summed up nicely, up until the last point. I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0. That said, I've tried. I reach…

They are fools. I will build you 1k servers with 100g to AWS in 75 days. Your $1M estimate is 3X too high at 2c/GB. In two hours I'll save you $25k per month. In 6 months $300k/month. Either you are growing and should be investing in cost efficiency or you run a staid lifestyle business. Which is fine, but if you aren't growing, get off expensive cloud.

You've a good logic in this, but you don't have enough details about us to judge it properly. The gain would be much smaller than you think and we would lose a lot of freedom, which would slow us down.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#159

Post by doh »

user5994461 wrote: post #151
doh wrote: post #145

Earlier quoted context omitted.

@user5994461 can't reply to your comment for some reason, but you're correct. Ansible would never scale. We ditched it a long time ago. We're using packer to create images/docker and then distribute them automatically via kubernetes or just directly through the custom images. As I've mentioned earlier, we're running an immutable infrastructure. That means once we need to change something we replace the whole server.…

The "reply" button sometimes goes away when the discussion is deep enough. Gotta click the comment to comment. I would imagine that rolling a container to thousands hosts may take a while. What kind of load and software do you run? In my experience, dramatically scaling out increases load and latency variance and cause all kind of problems. When was your experience with SoftLayer? Care to elaborate? I've got some int…

You're still thinking about these servers as a bare metal. We don't keep the servers running. We create a server always from scratch or based on an image we prepared or with a docker container that is automatically pulled from an internal repo once the OS is turned on.

This is the benefit of running on GCP. We don't have to trouble ourselves with the headache of scaling both with the images and containers thanks to the internal tools offered by GCP.

We quit Softlayer around February. We were running bare metal and they went down so often that we essentially ended up keeping one super large server for a very insignificant service. We never gave them too much of chance so I may be too harsh.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#160

Post by doh »

user5994461 wrote: post #152
late2part wrote: post #141

Earlier quoted context omitted.

It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minde…

> I think you're being close minded to building out a 20-60 cabinet data-center to offload some of your workload. I'd say it's time to forget about the infrastructure. Should optimize the software and rewrite in C++ :D

We already have huge portion of the system in C/C++ ;)
Post reply on HN