Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

thehftguy.com

Post reply on HN
Print view View original

141–150 of 179 posts

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#141

Post by late2part »

doh wrote: post #117
banachtarski wrote: post #60

Earlier quoted context omitted.

You should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

It's likely that a large percentage of your workload is static.

It's likely that bare metal is lower cost than spot - it is for me.

It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital.

I'm happy your company is doing great.

I agree GCP is a great competitor to AWS.

I think you're being close minded to building out a 20-60 cabinet data center to offload some of your workload.

It's hard to do a DC with over 20 racks and pay more than 1/2 of AWS's prices.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#142

Post by user5994461 »

geggam wrote: post #119
user5994461 wrote: post #108

Earlier quoted context omitted.

> Unless they're really bursty Last I checked. AWS has hourly billing and Google Cloud has sub-hour billing. If they are really bursty, they are in troubles with AWS pricing model. > the number where it makes more economic sense to build out your own data center. I would think that this number doesn't exist... unless you have already built datacenters for yourselves and you have a major internal expertise in that.

I can tell you less than 1000 servers in AWS runs around 200-300k US / month I am not sure what he is doing at 25k servers but you can see real world math puts this in a serious money range. Data centers can be built for less. I wonder if the down voters on the hardware comments actually run services at scale.

n1-standard-1 * 25k instances = $180k per month with pre-emptibles

You didn't see that coming, did you? :D

That gives you 25k CPU + 83TB of memory. (Of course, that's just to give a figure, they probably don't use single core instances).

Forget everything you know about AWS, it's over priced. Google is half the costs of AWS in average (down to one quarter for special cases).

I'm not sure what you mean by building your own datacenters (really, the buy some land and build from scratch?). Just having a bunch of dudes in a few locations worldwide is gonna be a multi-million dollar projects. And I'm not even talking about building, power, cooling, servers, storage, network. That's the hell of an undertaking.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#143

Post by user5994461 »

doh wrote: post #117
banachtarski wrote: post #60

Earlier quoted context omitted.

You should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

Would you mind giving more details about what you run with those 25k servers?

I'd worry about the time it takes to reconfigure these servers with ansible/salt. or the time it takes to kill/rebuild them from a system image.

On a side note, the 4th competitor is IBM SoftLayer. It's different enough from the top 3 (Amazon, Google, Azure) to be a thing of its own.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#144

Post by doh »

late2part wrote: post #141
doh wrote: post #117

Earlier quoted context omitted.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minde…

You summed up nicely, up until the last point.

I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0.

That said, I've tried. I reached out to many hosters, like OVH and others. They just don't have the capacity we need. 20 servers will not make a change for us. We wanted to start with 500, but it would take them 9 months.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#145

Post by doh »

doh wrote: post #117
banachtarski wrote: post #60

Earlier quoted context omitted.

You should have moved to bare metal months/years ago. Things like making your servers preemptible is like putting a bandaid on a gaping bullet hole.

There are many reasons why bare metal doesn't make much sense for us, at least at the moment. Here are couple: - we're very small team (13 people). We would never be able to manage this kind of infrastructure - we're growing incredibly fast. Just 6 months ago we were running on ~300 servers. We grew so quickly that we are now working with GCE's infrastructure planers to plan our resource needs - even we're not suffer…

@user5994461 can't reply to your comment for some reason, but you're correct. Ansible would never scale. We ditched it a long time ago. We're using packer to create images/docker and then distribute them automatically via kubernetes or just directly through the custom images.

As I've mentioned earlier, we're running an immutable infrastructure. That means once we need to change something we replace the whole server. Each server runs only one single service. That allows us to run smaller instances but in large quantities.

We actually did run on Softlayer. It was a nightmare. They have consistently some outage somewhere. We couldn't count on any instance to stay up. The performance was better, but you can't threat the infrastructure as a cattle and that was a huge limitation for us.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#146

Post by doh »

nodesocket wrote: post #137
doh wrote: post #134

Earlier quoted context omitted.

We don't host on AWS, we host on GCE. Also you're way off with your estimation. We pay significantly less. As the instance go, we have almost everything ranging from many thousands on the micro side to hundreds on the largest side.

$200 a server was my estimate which is pretty conservative. I assumed that price included disk and bandwidth per server. 25,000 * $200 = 5 million a month.

Well, no. You're calculating with 100% utilization + with normal servers. Preemptible (or spot) instances are way cheaper. For instance 8 core 7GB ram preemptible server on GCE costs around $45/month under full utilization. We may use it for couple of hours a day and then terminate and open a different one.

It truly differs minute by minute.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#148

Post by user5994461 »

saurik wrote: post #120
user5994461 wrote: post #107

Earlier quoted context omitted.

Honestly. I would ignore any comment that is that old. Running anything on AWS or Google 3 years ago. That must have been hell.

The attitudes and competencies of organizations generally doesn't change over the time scale of only a year and a half (which is really not long at all... I mean... I've been using AWS now for over eight years and had a product deployed on App Engine six years ago; the people I work with who really like GCP have been using it for three or four years, and our conversations about it and their experiences track that far…

Agree that the general criticism didn't change. (And the criticism from that link was on AppEngine, so that's certainly on point).

However 1-3 years is a big gap in terms of capabilities and tooling.

If I go back long enough ago, there was a time without nat instances, without VPC, without region in X zone where I have customers. That's deal-breaking changes. Some things we use today to manage our servers didn't exist or didn't support a specific usage some months ago.

I'm certainly biased because I've only been using AWS since last year. I see the things we do which are marked as "added MM-YY" and it would have been hell to try to do that 3 years ago.

That's why I advise to take old criticism about capabilities with a grain of salt.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#149

Post by late2part »

doh wrote: post #144
late2part wrote: post #141

Earlier quoted context omitted.

It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minde…

You summed up nicely, up until the last point. I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0. That said, I've tried. I reach…

They are fools. I will build you 1k servers with 100g to AWS in 75 days.

Your $1M estimate is 3X too high at 2c/GB.

In two hours I'll save you $25k per month. In 6 months $300k/month.

Either you are growing and should be investing in cost efficiency or you run a staid lifestyle business. Which is fine, but if you aren't growing, get off expensive cloud.

Re: GCE vs. AWS in 2016: Why You Shouldn't Use Amazon

#150

Post by late2part »

doh wrote: post #144
late2part wrote: post #141

Earlier quoted context omitted.

It's likely that a large percentage of your workload is static. It's likely that bare metal is lower cost than spot - it is for me. It's likely that your issue is allocation of capital - you'd rather put your money into burning dollar bills into AWS versus pre-paying for 18 months of servers with capital. I'm happy your company is doing great. I agree GCP is a great competitor to AWS. I think you're being close minde…

You summed up nicely, up until the last point. I understand why would think so, because you don't know our setup. Just to give you an idea, we process 16PB of data every single month. This is an ingress traffic. If we would pay for this traffic going out (egress), we would end up paying over $1M just for the traffic itself. By keeping everything in a single spot it costs us literally 0. That said, I've tried. I reach…

They are fools. I will build you 1k servers with 100g to AWS in 75 days.

Your $1M estimate is 3X too high at 2c/GB.

In two hours I'll save you $25k per month. In 6 months $300k/month.

Either you are growing and should be investing in cost efficiency or you run a staid lifestyle business. Which is fine, but if you aren't growing, get off expensive cloud.

Post reply on HN