Cutting down AWS cost by $150k per year simply by shutting things off • Hacker News Board

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#161

Post by jddj » Mon, Jan 22, 2024, 9:13 PM UTC

mlsu wrote: post #133

paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

Maybe there should be a sort of "anti-saas-sales" role: you get commission on whatever costs you're able to justify as superfluous. After all, the person at AWS makes commission selling you the stuff.

Funny downvotes. Although not structured like this value engineering is common in other areas.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#162

Post by Draiken » Mon, Jan 22, 2024, 9:31 PM UTC

killingtime74 wrote: post #142

devin wrote: post #131

Earlier quoted context omitted.

Unfortunately it seems like businesses only wake up and asks for cost reduction on the infrastructure spend side once the problem is out of control. At that point, the level of operations effort to get it under control feels more like a "big rewrite" than a collection of small tweaks.

It's opportunity cost. If it's not a problem then you can just make more money with your time.

Nah, that'd imply they think about this objectively. Most companies simply don't.

Most time this "opportunity cost" is then spent on useless hacky features that are never used and forgotten right after release (redesign anyone?).

Of course there are exceptions to all of these, but IME the majority of companies are either focusing on pennies or ignoring it completely. Not sure why you don't see balanced approaches more often. Maybe this will change with less VC money flying around.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#163

Post by avidiax » Mon, Jan 22, 2024, 9:41 PM UTC

jfim wrote: post #127

paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

Part of it is that it creates an incentive to create wasteful systems, only to "optimize" them later to rack up a bonus. Even if it gets changed to only pay out for reducing spend incurred by other engineers, it's possible to collude in such a way to extract bonuses from the company. A better way to have aligned incentives for the company and the employees would be to allocate a bonus pool for the entire company, fro…

> allocate a bonus pool for the entire company, from which AWS expenses are taken out of

Also a perverse incentive.

If we use ec2.small, the customer's query will take 3x longer but be half the price. Let's turn off the nightly security audits. We can live with quarterly backups, right? What do we need all these logs for, anyway? We could hack something that works together in 2 weeks, but if we spend 3 months, it could be really efficient, let's do that...

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#164

Post by whummer » Mon, Jan 22, 2024, 9:55 PM UTC

To give this a slightly different spin:

--> "The best optimization is simply not spinning things up!"

At least for local development and testing, as made possible by LocalStack (https://localstack.cloud), among other local testing solutions and emulators.

We've seen so many teams fall into the trap of "someone forgot to shut down dev resource X for a week and now we've racked up a $$$ bill on AWS".

What is everyone's strategy to avoid this kind of situation? Tools like `aws-nuke` (https://github.com/rebuy-de/aws-nuke) are awesome (!) to clean up unused resources, but frankly they should not be necessary in the first place...

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#165

Post by tayo42 » Mon, Jan 22, 2024, 10:06 PM UTC

paxys wrote: post #125

I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…

Companies make and spend so much money that this doesn't matter. Thousands, 10s of thousands, 100s is pointless in comparison the potential of building features. A developer costs about half million a year, (salary, bonus, rsu, taxes, benefits)

If they are paying you to saves 10s to maybe hundreds the company is losing money on you so they won't do this.

If your at a public company, look at your company's quarterly reports and see what it would take to many any kind of impact on net income.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#166

Post by gurchik » Mon, Jan 22, 2024, 10:09 PM UTC

Detrytus wrote: post #138

gurchik wrote: post #71

I recently helped save $150k per year by deleting node_modules. I noticed that one of our S3 buckets had high data transfer costs, a bucket that our app downloads HTML+JS assets from when we push out a new release. I downloaded the "directory" of files for our latest release and saw it was mostly node_modules. I checked the code and confirmed that, yes, if this file exists in the bucket then it'll be downloaded by th…

Now, the inconvenient question: how much of that $1m savings ended up in your bank account as a bonus? Because certainly some of it should :-D

Not sure yet, but probably nothing. I completely understand the expectations written in this thread to receive something in return, but I've given this thought and I'm not sure how to do this in a fair way in this situation. First, I was given dedicated time that quarter to work on cost savings and other people weren't, if I received a bonus is that fair to other people who didn't have the same opportunity? Not to mention the possibility of people abusing this process.

I would be happy to receive some extra cash, don't get me wrong, but I work for non-monetary benefits as well, and I have received some of those as part of this work. If I worked at a company with a different culture and I was being punished for doing the work, I would demand some bonus.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#167

Post by nvm0n2 » Mon, Jan 22, 2024, 10:16 PM UTC

anuptalwalkar wrote: post #17

Couple of years ago I saved about 14 mn of revenue per year for my company. I got a 250$ bonus for it.

But was that your job? Because if so, you really got salary+bonus. And if you'd found nothing you'd still have got salary. So you can look at it several ways.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#168

Post by nvm0n2 » Mon, Jan 22, 2024, 10:22 PM UTC

bdcravens wrote: post #108

tehlike wrote: post #100

Earlier quoted context omitted.

Hetzner cloud supports us-west and us-east.

I was looking at dedicated hardware. If I decide to stay in anyone's cloud, I'd stick with AWS.

There's alternative companies with similar offers, like Deft:

https://deft.com/dedicated-servers/

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#169

Post by nvm0n2 » Mon, Jan 22, 2024, 10:23 PM UTC

Chico75 wrote: post #117

ponector wrote: post #47

Earlier quoted context omitted.

Great! Shareholders will be happy. Did you receive any bonus?

The problem is always around abuse. If it becomes known that you can get a big bonus by wasting a lot of money on useless infra first and then reducing it, other people will start playing the game. How do you reward cloud cost awareness without creating perverse incentives?

It's always the same answer: managers who pay attention to the details. People familiar with your work should be able to tell if you're gaming the system or not.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#170

Post by cddotdotslash » Mon, Jan 22, 2024, 10:24 PM UTC

8organicbits wrote: post #112

cddotdotslash wrote: post #81

I’m convinced that once a company reaches ~$10m/year in AWS spend it becomes entirely reasonable to hire an in-house engineer whose sole job is to find cost savings opportunities. Literally a “find unused stuff and turn it off” engineer.

I've spent some time doing this. There's always old systems people don't really understand, ownership is poorly defined, and no one knows what happens if you turn it off. It's archeology. Understand what the system is doing and how it interacts with other systems and the business. If it looks unneeded back it up, stop the VM, wait and watch for fallout, and eventually terminate it.

There’s definitely a science to it. To complicate matters, the way you explore those connections, take backups, identify owners, and perform restores is different across pretty much every cloud service.