Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A million-dollar engineering problem

segment.com

Post reply on HN
Print view View original

231–240 of 265 posts

Re: A million-dollar engineering problem

#231

Post by patrickg_zill »

walrus01 wrote: post #87
Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

A million dollars buys a shitload of fast 1U dual socket xeon servers each with 512GB of RAM.

There are companies that will lease you a dedicated server with 40 cores and 256gb RAM for $499/month.

Figure a 10% quantity discount and you are looking at at least 200 of those for about $1 million per year.

Re: A million-dollar engineering problem

#232

Post by dexterdog »

patrickg_zill wrote: post #178
brianwawok wrote: post #164

Earlier quoted context omitted.

It's early defined in the docs. The machine also has a physical core count that is yours.

There is nothing stopping Amazon from having a machine that is double that config, and renting you half of it, however. As soon as 32GB DIMMs drop enough in price, I would expect them to do exactly that. You can configure a Dell R930 with e7 Haswell CPUs online, if you want to double-check it. Not sure if the R930 can go to 8 CPUs (which the e7 Haswells CPUs support).

But as soon as they double the size of the machine there is no reason for them to not have an instance type that is the size of the machine minus the host layer.

Re: A million-dollar engineering problem

#233

Post by dexterdog »

hueving wrote: post #200
illumin8 wrote: post #144

Earlier quoted context omitted.

This is just factually incorrect. AWS hard partitions all instance types, except T2 (which are overcommitted, and clearly advertised as such: "burstable"). So, if you provision an x1.32xlarge with 128 vCPU and 1.92TB of RAM, you get a single, dedicated host with that much CPU and memory. Nobody else gets it - it's dedicated to you 100%. The profit AWS is making is purely due to datacenter efficiency and being able to…

>The profit AWS is making is purely due to datacenter efficiency and being able to automate their operations at scale. That's what they want you to think, but a lot of it comes in the form of ripping you off for bandwidth to actually get data out of EC2.

I've been with AWS for about 10 years now on a number of projects. Bandwidth pricing has always been my major gripe. Almost everything else I consider fairly priced for what I get and how hands-off I get to be.

Re: A million-dollar engineering problem

#234

Post by mrlatinos »

rdtsc wrote: post #191
QuinnyPig wrote: post #51

Earlier quoted context omitted.

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

> I consult full time on AWS cost optimization / reduction / understanding. Curious, have you ever consulted a client not to use AWS? As in "your cost optimization is going to be not using AWS and using $x instead"?

Curious about this as well. Otherwise, it's like a car salesman saying you shouldn't buy a bike, rather, understand mpg before buying a car.

Re: A million-dollar engineering problem

#235

Post by patrickg_zill »

rbranson wrote: post #28
nik736 wrote: post #5

Have you guys considered going bare metal or a hybrid approach? With such immense spendings (even when saving the $1m/yr) it would probably be a lot cheaper.

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…

I would bet you 1000 USD that it would be eye-opening to buy a 64gb RAM, 8 core CPU server, put 4 SSDs in it in RAID10 config, and test your entire stack on just that box using Docker, KVM, whatever. Just running it beside your desk and having your other devs beat on it.

You might have to stub out some things using e.g. Cassandra or another open source alternative; just use a Docker or VM image without the production redundancy.

You did admit, that you have no idea how much performance is being dropped on the floor due to not using dedicated hardware. So why not test it?

My belief: you would find you are paying AWS far more than they are actually worth.

Re: A million-dollar engineering problem

#236

Post by nulagrithom »

kafkaesq wrote: post #116
QuinnyPig wrote: post #51

Earlier quoted context omitted.

> At this point simply not using AWS is a competitive advantage. Respectfully, I'm going to disagree. I consult full time on AWS cost optimization / reduction / understanding. If you blindly run things on AWS without an understanding of the costing model, that'll work for a time. As you scale, you start to realize "oh my god it runs on money." There are myriad ways around that, but a blanket "never use AWS" isn't goi…

A blanket "never use AWS" isn't going to address it constructively. But that's not what the commenter was saying, of course. More like (paraphrasing) "before you go full-hog on AWS, do some simple math first." Which an amazingly high percentage of people neglect to do, these days.

> before you go full-hog on AWS, do some simple math first

I had a coworker who was turned off of AWS by clicking the "Dedicated IP" checkbox when deploying a CloudFlare distribution with SSL.

One month and $600[0] later he swears up and down that AWS is the devil.

You can spend an amazing amount of money on AWS by accident and/or without need. Hourly pricing is a huge footgun. Best to run everything through the "Simple" Monthly Calculator: https://calculator.s3.amazonaws.com/index.html

0: https://aws.amazon.com/cloudfront/pricing/

Re: A million-dollar engineering problem

#237

Post by rbranson »

ig1 wrote: post #216
rbranson wrote: post #174

Earlier quoted context omitted.

That was definitely part of the effort; it just doesn't make for terribly compelling technical content. We have a healthy, active relationship with our AWS team. They worked with us to put in place minimum commits for several products, and examined both EDP as well as bulk RI purchase discounts.

I'd say especially in your case you could be pushing for additional discounts beyond what's available to everyone else. You're the ideal case for this, not only will you be a large customer in your own right if you succeed but you have a multiplier effect in that your own clients become heavier users of AWS because of your product encouraging users to move large amounts of data from Salesforce, etc. into AWS. I don't…

Some of the minimum commits aren't exactly standard. I'd talk more about them but they're covered by non-disclosure. I see what you mean by the multiplier effect increasing our bargaining position — totally makes sense.

Re: A million-dollar engineering problem

#238

Post by reacweb »

Negitivefrags wrote: post #16

A friend of mine was annoyed that a small service he liked was shutting down. He contacted the developer who said that they were shutting it down because the server costs were higher than the money they were making. They were spending 5k a month on AWS crap and claimed it was impossible to get any lower. He helped them consolidate everything onto a single rented dedicated server costing 400 a month. Now the service i…

IMHO, AWS is fabulous when your needs are a fast moving target. When you have something stable and have time to optimize for performance and cost, dedicated servers are better (cheaper and/or faster). There is no silver bullet.

Re: A million-dollar engineering problem

#239

Post by rbranson »

patrickg_zill wrote: post #235
rbranson wrote: post #28

Earlier quoted context omitted.

It could be if our workload was relatively stable and there were spare engineering cycles to undertake a migration and all that this entailed. Neither of these is the case. Much of what allowed us to implement these savings quickly with a small team was the flexibility afforded by cloud infrastructure. Poor decisions are easy to reverse, but in a bare metal world you better be damn sure what you're doing, which slows…

I would bet you 1000 USD that it would be eye-opening to buy a 64gb RAM, 8 core CPU server, put 4 SSDs in it in RAID10 config, and test your entire stack on just that box using Docker, KVM, whatever. Just running it beside your desk and having your other devs beat on it. You might have to stub out some things using e.g. Cassandra or another open source alternative; just use a Docker or VM image without the production…

I've operated on-prem and colo physical data centers. I've also been part of larger companies where someone else deploys baremetal on my behalf. I've been running infrastructure on top of AWS off-and-on since 2008. I've done a live migration of one of the largest Internet sites from AWS to baremetal in data centers. I feel pretty good about keeping our infrastructure in AWS for the foreseeable future.
Post reply on HN