Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We reduced our server costs by moving away from AWS

levelup.gitconnected.com

Post reply on HN
Print view View original

281–290 of 355 posts

Re: We reduced our server costs by moving away from AWS

#281

Post by antod »

jjav wrote: post #258
MajimasEyepatch wrote: post #125

Earlier quoted context omitted.

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

> have to hire But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away. There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people. At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scal…

Yup, I worked for a company that moved everything from colo to AWS. Not only did the annual tech costs increase about 10x (even with all the AWS migration credit funny math), but they ended up with double the number of engineers to look after it.

Admittedly this was hopefully only going to apply for a few years until they finished rearchitecting everything, but I doubt it would ever reduce back down to anywhere near the original costs.

The equation is also mirrored for something that started and grew on AWS though - going to bare metal means building up tooling and processes you didn't have before. The transition will be expensive in either direction.

Re: We reduced our server costs by moving away from AWS

#282

Post by icedchai »

dijit wrote: post #278
icedchai wrote: post #274

Earlier quoted context omitted.

I'm sure we all had different experiences with old school sysadmins. The ones I'm familiar with (early 90's, ISP industry) could do shell scripting and some perl. C was way out of their skill sets, except for a bit of copy-and-paste. The people who wrote tools like terraform and ansible were engineers with system administration skills. Those are very rare.

I’m biased. I worked in teams with such people basically my whole career. The pay increased, the titles and tools changed but the mentality didn’t. People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are. Sounds like they were automating though. “Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.

There was certainly some automation being done by the sysadmins I was familiar with (I did say "relatively little" in my original post, meaning compared to today!) It was definitely more pets than cattle at the places I was familiar with. These were small-ish companies, and the business didn't care as long as the systems worked.

Re: We reduced our server costs by moving away from AWS

#283

Post by dijit »

icedchai wrote: post #282
dijit wrote: post #278

Earlier quoted context omitted.

I’m biased. I worked in teams with such people basically my whole career. The pay increased, the titles and tools changed but the mentality didn’t. People just started beating their chest about devops and pooping on the legacy of sysadmins which is what most devops/SRE are. Sounds like they were automating though. “Cattle not pets” was a sysadmin mantra, but the business wanted pets most of the time.

There was certainly some automation being done by the sysadmins I was familiar with (I did say "relatively little" in my original post, meaning compared to today!) It was definitely more pets than cattle at the places I was familiar with. These were small-ish companies, and the business didn't care as long as the systems worked.

Sounds like a “devops” in that same position would be doing the same then.

Re: We reduced our server costs by moving away from AWS

#284

Post by oxfordmale »

xani_ wrote: post #268
mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

80k ? Honestly probably does . Our ops team of 3 spends maybe 10% of the time on the managing of few hardware racks we have in our local colocation. There are even months where nothing at hardware/hypervisor level is touched

3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.

Re: We reduced our server costs by moving away from AWS

#285

Post by jjav »

icedchai wrote: post #276
jjav wrote: post #263

Earlier quoted context omitted.

> Most old school sysadmins were doing everything manually with relatively little automation. That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there. The growth of perl, for example, was in great part from the sysadmin community automating everything.

Most of the scripts I was familiar with from that time were one-offs. The code wasn't very reusable. They were automating a task on one machine. Today's "DevOps" are automating things across N machines. It is a matter of scale.

> They were automating a task on one machine.

Remember that in those days you likely only had one machine. By "you" I mean the whole department. Everyone was logged into it and it handled email, talk, documents, compilation and debugging, etc.

That doesn't take away from the fact that the sysadmins were automating everything they needed to do, in the enviornment that existed.

Re: We reduced our server costs by moving away from AWS

#286

Post by jacobyoder »

bratao wrote: post #64
frognumber wrote: post #43

Earlier quoted context omitted.

In most businesses I've worked in: developer costs >> infrastructure costs An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense. PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops…

I consider this a fallacy. I´m on the dedicated server camp, but had to use AWS in a "everything in cloud" company. I spend way more time with Lambda/Cloud formation/ECS shenanigans than the terraform recipe I use in dedicated servers. And this is not even considering the higher latency between services and how hard is to debug it. In your case, you are not exchanging 2 hours of a developer for a higher bill. In my e…

FWIW, I'm in your camp mostly too.

That expert may not know what to do either, or... will contradict earlier folks and say "you need to redo all of this pile..." and you'll be stuck.

I've no doubt there are some workloads that really do require a level of complexity that various cloud systems offer. Much of what I've seen doesn't require it, but once someone starts down that road, they're 'justified' in learning more and tying more stuff to the cloud provider's way of doing things.

I cut my teeth setting up servers back in the 90s, and that you can spin up 'full' servers (thinking various shades of VPS) in a few seconds is crazy great. But we're somehow now 'past' that... and we have mountains more complexity to consider.

Re: We reduced our server costs by moving away from AWS

#287

Post by treffer »

agentultra wrote: post #272
moduspol wrote: post #214

Earlier quoted context omitted.

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is dra…

How many applications need infinitely scalable distributed object storage? I’ve worked at exactly one storage company that had a customer that had exascale data. They were doing cancer research as I recall and their test machines generated a lot of data. I heard stories about CERN at conferences but they also self host their data. But those were outliers. All of the large and small enterprises outside of that could f…

This is my favorite joke.

Is the issue big data or small machine?

Memory size of the largest AWS/GCP/... instances is a good indicator for small machine.

It doesn't mean I would go for the large machine. It just means I won't subscribe to "big data" as a reason to do X.

And it forces me to recalibrate this boundary regularly. Looks like double digit TB is currently the memory boundary for renting. I was still on the single digit TB train.

Re: We reduced our server costs by moving away from AWS

#288

Post by Draiken »

neeleshs wrote: post #246
boltzmann-brain wrote: post #187

Earlier quoted context omitted.

Indeed, and in fact running own metal is an order of magnitude easier than puzzling around the Brazil nightmare that AWS is. Both people cost money. It's not like if you go with AWS things run themselves.

The value prop of AWS/GCP etc go beyond bare metal vs VMs. Running HA databases (even if you are a startup, you need this), centralized logging, secrets manager, KMS, making sure disks are encrypted, something like pub/sub as a messaging backbone for your application, load balancers... the list goes on. Huge difference b/w reading a few pages of documentation on secrets manager, and clicking a button to get the servi…

Yet most of these aren't needed for small crews... do I care that my DB is not HA when I have 1 RPS? You said yes, but I disagree. Do I need centralized logging if I have a few servers? It's already centralized for free.

It's much easier to setup and forget basic bare metal servers with PG/NGINX and whatnot, than it is to automate using dozens of AWS services.

People pretend that AWS doesn't cost engineers to run it, when it's IMO basically the same human cost, if not bigger as complexity grows. You just don't pay that cost upfront, but you sure do pay it later with interest.

You get stuff like HA but that's not free. You also now have to manage a new boatload of services, scripts, changing APIs, etc.

Re: We reduced our server costs by moving away from AWS

#289

Post by dijit »

lakomen wrote: post #244
moduspol wrote: post #214

Earlier quoted context omitted.

If that were true, then there'd be no value proposition to AWS. It is absolutely easier to use S3 than to create your own fast, highly available, infinitely scaling storage solution on your own metal. It requires more than zero knowledge / expertise to use S3, but far less than it would to implement and run yourself. If you can accept that, then we already agree in principle. It's just matter of where the line is dra…

Minio is FOSS and a lot cheaper. I don't see your point

I think that's a misrepresentation of the argument though.

MinIO by itself is fantastic software for sure, but running storage appliances is shades of difficult or expensive.

On the one hand you can buy a netapp filer and support plan which is basically as self-healing as the cloud is, netapp will send a human to go replace failed drives before you even know a fault is imminent.

On the other, that's expensive but running your own is complex if you're not setup to do it already.

MinIO is but one part of a data storage puzzle. Though a very important one.

Re: We reduced our server costs by moving away from AWS

#290

Post by [deleted] »

speedgoose wrote: post #252
twawaaay wrote: post #223

Earlier quoted context omitted.

It is a tradeoff. Whatever you choose will have its costs. You need to know the tradeoff you are making but what I am saying is that, in most cases, it costs more to spend so much effort up front to be "cloud agnostic" than any benefits of maybe being able to switch cloud environment in a hurry.

I agree. I have switched cloud environments at work in the past, and it required a lot of efforts. However we switched towards standard Kubernetes to have more flexibility in the future. Also sometimes you don't want to minimise the cost but use the best services or reduce the risks. Let's say I go full on Azure, but after a year or two GCP offers a much better product for my use case. I will have to ignore it becaus…

[deleted]
Post reply on HN