Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

211–220 of 355 posts

Re: We reduced our server costs by moving away from AWS

#211

Earlier quoted context omitted.

How much of your developer time gets lost to maintaining servers. I used to think this sort of thing was a valuable use of my time but I now have over $200k in annual revenue and we pay less that $5 a month in raw compute (and that's nearly all S3). My Co-founder is always worried about the cost of our AWS but so far we just haven't witnessed it. My mandate has always been the price should be able to scale to 0. So n…

The more I work with cloud the more I see this 'saves time on infrastructure' being a half-truth at best - we still have overwhelmed DevOps people who, unlike managing on-prem services, have 0 insight or control over what goes on inside AWS. And the services require a decent amount of hand holding, proprietary know how, etc. In the end you replaced sysadmin with DevOps and got up charged multiples.

It's certainly possible to overspend on cloud services but in most cases when I see comparisons people tend to forget to fully include their true costs for things like staff time, infrastructure, etc. and especially things like opportunity cost for the delays caused by provisioning infrastructure, less capable interfaces (e.g. if you're swapping Terraform for a Jira ticket), or the technical decisions people make because they have fewer services available (this could be a Lambda function but we don't have that so now we need to manage a full VM).

Some examples which come to mind:

* Comparing S3 to the on-premise tape system but ignoring the fact that it involved an expensive tape robot, DR had access times measured in days, etc.

* Comparing S3 to on-premise storage, ignoring the difference in redundancy, forcing users to handle bitrot at the application level, and the procurement process meaning that when they ran out of storage it took months of telling people they couldn't allocate more.

* Saying their devops engineer cost twice as much as their sysadmins (true) but then when you look the devops engineer is using automation and managing literally a hundred times more systems than the “cheaper” ops team.

* Saying their cost to run a VM was cheaper than EC2, which true if you looked only at the instance but not when you calculated how much they were spending on underutilized VM hosts, power / HVAC, facilities people, etc.

It's totally possible to beat a major cloud provider on costs[1] but you usually need to be operating at fairly large scale to even approach the break-even point. This is especially true when you have regulatory or policy requirements for things like security and you include the cost of monitoring all of the things which fall under the cloud provider's responsibility — management networks, firmware version management, robust logging and IAM, etc. are all easy to accidentally exclude when making comparisons.

1. Network egress as the most obvious area to attack

Re: We reduced our server costs by moving away from AWS

#212
post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

Exactly. In the "old IT world" we call this TCO = Total Cost of Ownership.

Re: We reduced our server costs by moving away from AWS

#213

Earlier quoted context omitted.

huge salaries + ten "ops engineers", lol. I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu

One time I was doing research on some really cheap data centers in basically 3rd world countries. Just out of curiosity to see how cheap it could really get. One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture? I th…

I mean, the one I'm talking about is relatively reasonable on infra

they've decent building, data center tier, so power/network redundancy, etc, etc

they're just cheap as fuck when it comes to people

Re: We reduced our server costs by moving away from AWS

#214

Earlier quoted context omitted.

AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

If that were true, then there'd be no value proposition to AWS.

It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself.

If you can accept that, then we already agree in principle. It's just matter of where the line is drawn for various services and use cases.

Re: We reduced our server costs by moving away from AWS

#215
post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Anecdotal, but for one of my projects, Google Cloud / Compute Engine VMs cost around ~$5k a month all in. The exact same setup, when we moved it to LiquidWeb, cost us $2k.

Don't underestimate the savings that can be made from switching from a big-name cloud provider to a more old school hosting provider.

Re: We reduced our server costs by moving away from AWS

#216
This is unsurprising.

The point of AWS is to be flexible. You’re paying for that. It’s easy to start. It’s easy to stop. It’s easy to change capacity.

Running your own servers is none of these things. But it is cheaper at sufficient scale. You can’t ignore the labor cost (particularly engineering) however.

Where AWS shines is with highly volatile workloads. With your own servers you have to provision for peak capacity. That’s less the case with AWS.

No shade on the author of course. It’s great to read things like this.

Re: We reduced our server costs by moving away from AWS

#217
post #64

Earlier quoted context omitted.

In most businesses I've worked in: developer costs >> infrastructure costs An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense. PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops…

I consider this a fallacy. I´m on the dedicated server camp, but had to use AWS in a "everything in cloud" company. I spend way more time with Lambda/Cloud formation/ECS shenanigans than the terraform recipe I use in dedicated servers. And this is not even considering the higher latency between services and how hard is to debug it. In your case, you are not exchanging 2 hours of a developer for a higher bill. In my e…

You could say this about... pretty much anything you're not well-versed in.

Trying to use a thing that accomplishes the a similaroutcome but in a totally different way requiring a totally different skillset, it's not surprising that you found yourself needing someone who does have that skillset to assist.

A Linux server guy trying to manage Windows servers will probably need help from a Windows server guy. A car driver trying to fly a Cessna somewhere will probably need help from a pilot.

Re: We reduced our server costs by moving away from AWS

#218
post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

People don't consider productivity? Maybe things have gotten a lot better in the industry now. Otherwise, to rehash an older comment on HN:

I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up and what the funk is eBPF? Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.

Re: We reduced our server costs by moving away from AWS

#219

Earlier quoted context omitted.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is dra…

Not necessarily. Probably they already have pool of admins/devops that ran their systems for past 30 years, but have 0 AWS experience. Also, most companies don't need to scale infinitely, not everybody is building next google (in fact, almost nobody outside SV is).

Re: We reduced our server costs by moving away from AWS

#220

Earlier quoted context omitted.

The alternatives are tremendously more expensive in both time and money (and I’m thinking engineer salaries etc. here, not just monthly service costs). For most businesses the savings in money and time are worth the risks you pointed out here. If those risks are too high for you (and fair enough if they are), you’ll pay one way or another to avoid them.

Is your cloud ops department free? I have personally seen that amazing sysadmins are a lot cheaper than passable cloud ops people.

I don't think that's universally true. But even if it were, you'd need far more people if you were to manage the infrastructure yourself than to just use AWS.
Post reply on HN