We reduced our server costs by moving away from AWS • Hacker News Board

Re: We reduced our server costs by moving away from AWS

#261

Post by MajimasEyepatch » Wed, Sep 28, 2022, 8:18 PM UTC

tester756 wrote: post #201

MajimasEyepatch wrote: post #125

Earlier quoted context omitted.

Thank you for bringing up the engineering cost. People always look at this as just AWS > Bare metal or whatever, but there's so much more to it than that. If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year, then they actually saved $400k. Which is still substantial and, all else being equal, sounds worthwhile. If they saved $800k per year, and they…

huge salaries + ten "ops engineers", lol. I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu

And there's no way in hell I'd trust them to run the core infrastructure for a business worth hundreds of millions of dollars.

Re: We reduced our server costs by moving away from AWS

#262

Post by Melatonic » Wed, Sep 28, 2022, 8:18 PM UTC

hintymad wrote: post #218

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

People don't consider productivity? Maybe things have gotten a lot better in the industry now. Otherwise, to rehash an older comment on HN: I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was ba…

That also just sounds like Uber had hired crap talent....

Re: We reduced our server costs by moving away from AWS

#263

Post by jjav » Wed, Sep 28, 2022, 8:19 PM UTC

icedchai wrote: post #241

dijit wrote: post #236

Earlier quoted context omitted.

There is a value prop for sure. It’s hard to quantify how best you’ll be served but a lot of people are following the mantra of “nobody got fired for going AWS”. It makes sense for some people, others are cargo-culting; yet more are fanning the flames of that cargo cult because their pay check depends on it. Sysadmins are/were paid much less than cloud native devops people, and you need the same number of them unless…

One good DevOps person can achieve much more than a single sysadmin. Most old school sysadmins were doing everything manually with relatively little automation. You are certainly correct about the the overly complex AWS reference architectures. I've seen relatively simple applications with just as much infrastructure code (generally "terraform", occasionally CloudFormation JSON) as application code. It's crazy.

> Most old school sysadmins were doing everything manually with relatively little automation.

That's not true of any place I experienced in the early to late 90s. If you meant earlier, perhaps, I wasn't there.

The growth of perl, for example, was in great part from the sysadmin community automating everything.

Re: We reduced our server costs by moving away from AWS

#264

Post by MajimasEyepatch » Wed, Sep 28, 2022, 8:19 PM UTC

taylodl wrote: post #231

MajimasEyepatch wrote: post #72

Earlier quoted context omitted.

I think that statement about EC2 is a little too strong. You definitely shouldn't be managing individual EC2 instances in most cases, and you probably shouldn't be deploying directly to EC2. But if you're running EKS (Kubernetes) or ECS (non-k8s containers), then you're probably going to get more bang for your buck with EC2 nodes than Fargate nodes, especially if you have a large cluster.

That's a good distinction. Maybe instead of saying avoid EC2 what we should be saying is try to utilize AWS' managed services whenever possible.

Yup, that's generally true. Otherwise, you really might not be better off with AWS.

Re: We reduced our server costs by moving away from AWS

#265

Post by pwinnski » Wed, Sep 28, 2022, 8:21 PM UTC

60secs wrote: post #180

pwinnski wrote: post #116

Earlier quoted context omitted.

> Ok so you save money on the monthly bill Right, you understand, but maybe you don't really understand? If you work at a company large enough, and with enough DevOps, NetOps, Ops people to go it alone, and can get favorable lease terms on hardware, then by all means, host it yourself! Paying all of those people and also paying for AWS seems odd, for sure. But if you work at a company without a deep bench in various…

If you need to hire 5-10 engineers to save $1MM, you didn't save anything.

Right, which is why I can't imagine moving from AWS to self-hosting. Having to hire a full set of Ops people to manage what AWS is managing for me would hurt.

Re: We reduced our server costs by moving away from AWS

#266

Post by Melatonic » Wed, Sep 28, 2022, 8:21 PM UTC

Corrado wrote: post #238

hamandcheese wrote: post #163

Earlier quoted context omitted.

> At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. Co-locating has no capital investment other than hardware, and is pretty cheap. A 40U rack of compute charged as equivalent ec2 instances has a retail price easil…

I feel that a lot of posts like this might be under-representing the true costs of running your own hardware. I was only tangentially associated with a large-ish operation and I can tell you that there are loads of things that take a lot of time but are often over looked. Things like detecting and replacing bad hardware. HDs don't last forever and when they go bad it's not fun; especially if you have to source some s…

Your VAR should be doing a lot of that work for you - sure you need to understand licensing but if you have a good rep they should be providing you with up to date info on all of this.

HDD replacement is trivial and some COLOs can even do it for you - and any good vendor will have a thing setup where your appliance automatically notifies them and a new drive is overnighted to your COLO. Sure - maybe a few times a year you drive out to swap a drive.

If you do not have good monitoring setup then that is entirely fixable - there are many stellar solutions these days out there. Hardware is easier than ever.

Re: We reduced our server costs by moving away from AWS

#267

Post by MajimasEyepatch » Wed, Sep 28, 2022, 8:21 PM UTC

tester756 wrote: post #213

citizenpaul wrote: post #203

Earlier quoted context omitted.

One time I was doing research on some really cheap data centers in basically 3rd world countries. Just out of curiosity to see how cheap it could really get. One of the companies had a picture of one of their "datacenters." It was something like 10 racks in a moldy unfinished basement with visible water on the floor of what I'm guessing was a residential building. Maybe they had mopped the floor for the picture? I th…

I mean, the one I'm talking about is relatively reasonable on infra they've decent building, data center tier, so power/network redundancy, etc, etc they're just cheap as fuck when it comes to people

People are the most important part of any operation! That's the last thing you want to cheap out on.

Re: We reduced our server costs by moving away from AWS

#268

Post by xani_ » Wed, Sep 28, 2022, 8:23 PM UTC

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

80k ? Honestly probably does. Our ops team of 3 spends maybe 10% of the time on the managing of few hardware racks we have in our local colocation. There are even months where nothing at hardware/hypervisor level is touched

Re: We reduced our server costs by moving away from AWS

#269

Post by jacobsenscott » Wed, Sep 28, 2022, 8:24 PM UTC

systemvoltage wrote: post #167

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

The initial bit is important though. It creates a circular dependency. If you start out without AWS, your entire company's software from how you build monoliths/microservices/queues changes. Look at Stack Overflow's architecture which stands apart because it was never designed to work in cloud from the beginning: https://stackexchange.com/performance I'd argue that 90% of the SaaS doesn't have SO's scale. The whole t…

Yes - but the FOMO industry is too strong.

Re: We reduced our server costs by moving away from AWS

#270

Post by jjav » Wed, Sep 28, 2022, 8:25 PM UTC

lbriner wrote: post #124

mabbo wrote: post #118

I'll always celebrate stories like this, but I also don't take some kind of anti-AWS lesson from it. This company saved $800k/year. Perfect time to go in-house with this solution. But when they were 1/10th this size, they'd only have saved $80k/year. Does that cover the cost of the engineering to build and maintain this system? Maybe not. And when they were 1/100th the size, it would have been laughable to go in-hous…

Yes, exactly this. At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily. How many people are now needed to fault-find bad hardware/software/networks, to be on call for any problems? How many calls out to the Electricia…

> fix some power issue

> large air-con system

You wouldn't usually jump from AWS to buying up real estate to build your own physical data center.

A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.