We reduced our server costs by moving away from AWS • Hacker News Board

Re: We reduced our server costs by moving away from AWS

#181

Post by boltzmann-brain » Wed, Sep 28, 2022, 5:36 PM UTC

hintymad wrote: post #168

joshstrange wrote: post #48

They don't mention at all what services they were using (other than slight mention of S3) which makes it very hard to respond to this. If you are running everything on EC2 then you are going to have a bad time (especially if you aren't using reserved instances). AWS (IMHO) shines with the various services they provide (S3, Lambda, CloudFront, API Gateway, SQS< SES, to name a few). AWS is a game of trying to reduce yo…

And likely a company who does not use cloud resorts to Chef/Puppet/TFE and what not to manage their machines. Such scripting tools may work for a small team as the engineers deal with their specific needs day in and day out, but it is expensive to scale out to larger teams. I'd assume only a few engineers would enjoy writing hundreds if not thousands of lines of yaml or the whatever so-called DSLs offered by the afor…

There's a lot of other businesses that blindly do cloud for no good reason at all. I calculated the forecast of aws vs own infrastructure (aws is what was being used) for an ML company doing major compute backfills. I did that at the height of the GPU price craze and the whole move would have still amortized itself after just 6 months. Why people stick to aws for that sort of stuff is beyond me. If you ever need to spin up VMs on aws rather than on your own metal, you can always do it then. It boggles the mind that people want to throw away millions to get worse performance and higher error rates.

Re: We reduced our server costs by moving away from AWS

#182

Post by pclmulqdq » Wed, Sep 28, 2022, 5:36 PM UTC

pwinnski wrote: post #169

pclmulqdq wrote: post #66

Earlier quoted context omitted.

At this point, it should be pretty clear that self-hosting is cheaper, given all of the examples we have seen. AWS's economies of scale have as much to do with its pricing as the phases of the moon, and citing them as a reason why AWS could be cheaper than self-hosting is pretty ignorant in itself.

On the contrary, self-hosting is only cheaper under some circumstances, which is why when those circumstances are met, it becomes a story worth a post on HN. Whether economies of scale, or the efficiency of managed services at scale, or use-based loss leaders, there are many ways in which AWS services could be, and often are, cheaper than self-hosting. Of course, not always.

In my experience, it is almost always not cheaper to use AWS services, unless your workload is exceptionally bursty in an unpredictable way or fits entirely in the free tier. Pretty much nothing at AWS is a loss-leader except the free tier. Also, if your workload is small and you don't need to hire cloud ops folks, you can come out ahead on TCO.

Re: We reduced our server costs by moving away from AWS

#183

Post by throwaway894345 » Wed, Sep 28, 2022, 5:37 PM UTC

datalopers wrote: post #107

andy_ppp wrote: post #100

Earlier quoted context omitted.

Are you including the wages of the extra people to support building and managing such services?

extra people? no because I hire generalists actually capable of building software, not overpaid kids only capable of gluing aws services together.

You still have to pay your generalists for the time they spend writing and maintaining that bespoke software, and moreover you can hire competent developers to develop using cloud services. There’s no reason to assume that cloud services require incompetent developers, and indeed the cheapest solution is often to employ competent developers to integrate cloud services (whose development and operations costs are shared among all customers, whereas your company is footing the whole bill for your bespoke services).

Re: We reduced our server costs by moving away from AWS

#184

Post by otabdeveloper4 » Wed, Sep 28, 2022, 5:37 PM UTC

MajimasEyepatch wrote: post #125

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

Hiring a person to do 'docker compose up' for you is orders of magnitude cheaper than whatever AWS-specific knowldege is needed to not have AWS crap its bed.

Re: We reduced our server costs by moving away from AWS

#185

Post by zo1 » Wed, Sep 28, 2022, 5:38 PM UTC

moonchrome wrote: post #151

etothepii wrote: post #121

Earlier quoted context omitted.

How much of your developer time gets lost to maintaining servers. I used to think this sort of thing was a valuable use of my time but I now have over $200k in annual revenue and we pay less that $5 a month in raw compute (and that's nearly all S3). My Co-founder is always worried about the cost of our AWS but so far we just haven't witnessed it. My mandate has always been the price should be able to scale to 0. So n…

The more I work with cloud the more I see this 'saves time on infrastructure' being a half-truth at best - we still have overwhelmed DevOps people who, unlike managing on-prem services, have 0 insight or control over what goes on inside AWS. And the services require a decent amount of hand holding, proprietary know how, etc. In the end you replaced sysadmin with DevOps and got up charged multiples.

Agreed, devops (and whatever the hell cloud wranglers are being sold as today) is the new priesthood. It's meta-level job creation and job security.

Re: We reduced our server costs by moving away from AWS

#186

Post by jacooper » Wed, Sep 28, 2022, 5:39 PM UTC

theptip wrote: post #144

Perhaps I'm missing it in the OP -- I don't see any mention of what they actually moved to. CoLo? VPS? On-prem? This seems like a key detail when telling people about your migration off AWS.

On-prem

Re: We reduced our server costs by moving away from AWS

#187

Post by boltzmann-brain » Wed, Sep 28, 2022, 5:39 PM UTC

Dma54rhs wrote: post #161

MajimasEyepatch wrote: post #125

Earlier quoted context omitted.

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

Re: We reduced our server costs by moving away from AWS

#188

Post by boltzmann-brain » Wed, Sep 28, 2022, 5:41 PM UTC

otabdeveloper4 wrote: post #184

MajimasEyepatch wrote: post #125

Earlier quoted context omitted.

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

Hiring a person to do 'docker compose up' for you is orders of magnitude cheaper than whatever AWS-specific knowldege is needed to not have AWS crap its bed.

not just "not have AWS crap its bed", but whatever sacrifices you need to perform in order for amazon not to come back with a bill that just so happens to be $1M higher than you expected it to be because of some gotcha that amazon profits off of, like (say) routing stupid traffic to one of their servers, currently occupied by you, that would be idle otherwise. Nice "mistake" to make.

Re: We reduced our server costs by moving away from AWS

#189

Post by guhidalg » Wed, Sep 28, 2022, 5:41 PM UTC

galaxyLogic wrote: post #176

guhidalg wrote: post #170

Earlier quoted context omitted.

But why would you try to build something cloud-agnostic to run on the most cloud-aware tier: serverless? Computers don't come forth from the ether, your code is always running _somewhere_ and serverless means your provider makes a ton of choices for you. The lowest common denominator across clouds is a VM. If you can run your applications and your databases on VMs, you can run them on any cloud AND on-premises. If yo…

I wonder could micro-services be the solution? Run some of your micro-services on AWS and another on Azure etc. A given micro-service then might be highly dependent on AWS but the part of your application that uses those micro-services would not know anything about internals of AWS etc.

Perhaps if you're trying to get the best products from each cloud, but I doubt this would be worth it. For each cloud you are going to need to establish: accounts, payment, access control, continuous deployment, geographical locality of their DCs, distributed logging, billing (and watch out for egress costs!), etc... I cannot imagine the added complexity of doing this would overcome the benefits of doing everything with one provider.

But hey if your goal is to bill a client as much as possible by claiming that you're using the best products from each cloud, this sounds like a GREAT business idea.

Re: We reduced our server costs by moving away from AWS

#190

Post by boltzmann-brain » Wed, Sep 28, 2022, 5:42 PM UTC

hamandcheese wrote: post #163

lbriner wrote: post #124

Earlier quoted context omitted.

Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electricia…

> At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easil…

In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.