Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

Post reply on HN
Print view View original

251–260 of 355 posts

Re: We reduced our server costs by moving away from AWS

#251

Post by pclmulqdq »

JohnHaugeland wrote: post #210
pclmulqdq wrote: post #137

Earlier quoted context omitted.

R2 is an object store. It is literally a drop-in replacement to S3. Cloudflare is a CDN, which, like all of its competitors, is a lot cheaper than doing it yourself because CDNs are a competitive market . There are at least 10 (if not more) CDN providers out there that have footprints in several thousand datacenters, and the product is completely undifferentiated. In a competitive market, you have to charge related t…

> R2 is an object store. It is literally a drop-in replacement to S3. A CDN is a thing where you can put things, and request them by web, and have them geo-distributed. R2 is actually a CDN. So is S3. Yes, I'm aware, the vending companies also sell different products called CDNs. Nonetheless, R2 and S3 (unlike B2) are CDNs. You can make distributed hits to it across the planet to addresses you don't control. You can…

R2 is not globally distributed and neither is S3. All of my S3 buckets are in US-East. See here: https://community.cloudflare.com/t/cloudflare-r2-doesnt-dist...

R2 is not a CDN. S3 is not a CDN. Both are object stores. They store objects with some redundancy in a local area. CDNs are globally distributed. Object stores usually aren't. You can use an object store for backups and for holding webpages, but that doesn't make them backup services or webservers either. Using only S3 (without deduplication or compression) for backups is ridiculously expensive, and using it as a webserver limits you to a static site.

In terms of the costs, if you are small, the free tiers (and cloud platforms in general) are great. If you are at the point where you are paying $5-10k/month, you are almost certainly overpaying by using cloud services.

Also, you don't even need an object store for a small app. You can use a server with a hard drive if you have a lot of data to store (plus backups and a redundant server in another place for HA). Put a CDN in front of it - a real CDN - and don't worry about it. Pay the monthly cost for the CDN. Also pay for your backups.

It's fine if you want to pay AWS so that you can focus on things other than efficiency. Most SaaS companies make that trade. But you are paying for it.

Re: We reduced our server costs by moving away from AWS

#252

Post by speedgoose »

twawaaay wrote: post #223
speedgoose wrote: post #153

Earlier quoted context omitted.

Not being cloud agnostic forces you to use one single cloud provider, that is not necessarily the best in everything.

It is a tradeoff. Whatever you choose will have its costs. You need to know the tradeoff you are making but what I am saying is that, in most cases, it costs more to spend so much effort up front to be "cloud agnostic" than any benefits of maybe being able to switch cloud environment in a hurry.

I agree. I have switched cloud environments at work in the past, and it required a lot of efforts. However we switched towards standard Kubernetes to have more flexibility in the future.

Also sometimes you don't want to minimise the cost but use the best services or reduce the risks. Let's say I go full on Azure, but after a year or two GCP offers a much better product for my use case. I will have to ignore it because it's too expensive to switch. Or if I go full on GCP/Firebase but suddenly the pricing model changes and it becomes overpriced. I will have to eat the cost for some time.

However, if I'm careful to avoid vendor lock-ins from the start, it will probably cost a bit more on average, but the maximum cost is much lower and I don't risk being stuck with a bad service.

In practice I will for example avoid to write code specific to the proprietary Azure Blob Storage API, but use an S3 compatible object storage instead. I will also rather have an abstraction layer than using AWS SQS or GCP pub/sub directly.

Re: We reduced our server costs by moving away from AWS

#253

Post by dijit »

icedchai wrote: post #241
dijit wrote: post #236

Earlier quoted context omitted.

There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless…

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

You’re quite mistaken on the first point.

Most automation that you know of as “devops tools” are borne from sysadmins.

Terraform was written by a sysadmin; puppet, ansible, saltstack and cfengine too.

It is revisionist to think sysadmins were not automating their jobs.

Old school sysadmins used to know C, bash and Perl.

New school devops just traded that for a handful of DSLs and Python.

I’m sure some technicians working the IT department managed to get by running scripts created by sysadmins and claiming not to code, but it was definitely the common case that 20 years ago sysadmins could code and worked tirelessly to “automate themselves out of a job” (literally a mantra i was told as a sysadmin 15 years ago)

Re: We reduced our server costs by moving away from AWS

#254

Post by [deleted] »

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

[deleted]

Re: We reduced our server costs by moving away from AWS

#255

Post by pwinnski »

pclmulqdq wrote: post #182
pwinnski wrote: post #169

Earlier quoted context omitted.

On the contrary, self-hosting is only cheaper under some circumstances, which is why when those circumstances are met, it becomes a story worth a post on HN. Whether economies of scale, or the efficiency of managed services at scale, or use-based loss leaders, there are many ways in which AWS services could be, and often are, cheaper than self-hosting. Of course, not always.

In my experience, it is almost always not cheaper to use AWS services, unless your workload is exceptionally bursty in an unpredictable way or fits entirely in the free tier. Pretty much nothing at AWS is a loss-leader except the free tier. Also, if your workload is small and you don't need to hire cloud ops folks, you can come out ahead on TCO.

So we have gone from a categorical "not cheaper" to "almost always not cheaper," except for two circumstances, no, three. Progress!

At my last company, providing SaaS for the education market, moving from a datacenter to AWS saved almost 70% year-over-year. In the datacenter, we ran machines to cover our peak load, which only happened a few times a year. In AWS, we scaled way down and auto-scaled up during those peak weeks.

Is the entire education market "exceptionally bursty?" I suppose it could be considered so. Bar exams and midterms and finals certainly don't happen every week.

I'll wait for people to tell me how AWS was the wrong solution, and how we did everything wrong before that, but the bottom line is we saved a lot of money, accelerated our development schedule by building all new functionality using the so-called "serverless" stack, and succeeded so well during the pandemic that another company acquired us... which is why it's my former company.

Re: We reduced our server costs by moving away from AWS

#256

Post by late2part »

moduspol wrote: post #214
boltzmann-brain wrote: post #187

Earlier quoted context omitted.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is dra…

except that s3 doesn't infinitely scale.

Try uploading 5PB of content and downloading it that day at 1M RPS.

Re: We reduced our server costs by moving away from AWS

#257

Post by MajimasEyepatch »

neeleshs wrote: post #246
boltzmann-brain wrote: post #187

Earlier quoted context omitted.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

The value prop of AWS/GCP etc go beyond bare metal vs VMs. Running HA databases (even if you are a startup, you need this), centralized logging, secrets manager, KMS, making sure disks are encrypted, something like pub/sub as a messaging backbone for your application, load balancers... the list goes on. Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the servi…

Not to mention compliance. If you have to go through third-party audits (and most businesses do after reaching a certain size), then using standard services from a public cloud provider can simplify things significantly. Sometimes that's because you can just offload "security of the cloud" to the cloud provider, and other times it's because there are well-established guidelines for how to achieve, say, PCI compliance on AWS.

Re: We reduced our server costs by moving away from AWS

#258

Post by jjav »

MajimasEyepatch wrote: post #125
mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

> have to hire

But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away.

There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people.

At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.

Re: We reduced our server costs by moving away from AWS

#259

Post by Melatonic »

icedchai wrote: post #241
dijit wrote: post #236

Earlier quoted context omitted.

There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless…

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

Not necessarily true - many were doing tons of scripting.

Re: We reduced our server costs by moving away from AWS

#260

Post by pclmulqdq »

pessimizer wrote: post #245
pclmulqdq wrote: post #66

Earlier quoted context omitted.

At this point, it should be pretty clear that self-hosting is cheaper, given all of the examples we have seen. AWS's economies of scale have as much to do with its pricing as the phases of the moon, and citing them as a reason why AWS could be cheaper than self-hosting is pretty ignorant in itself.

Again. This was a reply to a strong, clearly expressed claim. You can make up other related claims to argue with, but you should find somebody actually making them in order to be more effective. It is not obvious that growing your own wheat will always be cheaper than buying bread.

It's interesting that you chose a competitive market to compare this with. In a competitive market, efficiencies go to the customer. In an oligopoly or monopoly, efficiencies go to the provider.

It's obvious that baking your own bread is a bad idea if you're okay with what you can buy from the competitive market. It's equally obvious that baking specialty, artisanal bread (and buying the commodity ingredients, such as flour, salt, and yeast) is cheaper than going around to several grocery stores to try to find the exact kind of loaf you want.

AWS is in an oligopoly, so it should be obvious that its efficiencies (in this case, its economies of scale) are not passed on to the customer. The strong, clearly expressed claim was fine.

Post reply on HN