Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

111–120 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#111

Post by hibikir »

VoodooJuJu wrote: post #96
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

And that's how insignificant the costs of cloud providers are in the grand scheme of things. It's a lot of money to a bootstrapping startup, but for the vast majority of these cloud providers' customers, it's a rounding error that's easily forgotten for 3 years. And that's precisely why you and your little bootstrapper or indie firm should not be using globocloud: you do not have mountains of cash to piss away. Bare…

You'd be surprised. I've seen AWS bills well in the 9 figures. It's just that fixing expensive designs is, in itself, quite expensive, and many of those very large corps have hiring practices that don't allow them to complete for the top of the market. Sometimes there's tens, if not hundreds of millions in savings a year, but corporate sclerosis makes it very difficult for broad cost-saving initiatives to be identified and approved.

It's the same issue in any large organization: Large levels of success somewhere allow for large levels of waste somewhere else, but often the waste is not required for the success to exist: The success just makes the organization complacent.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#112

Post by 8organicbits »

cddotdotslash wrote: post #81

I’m convinced that once a company reaches ~$10m/year in AWS spend it becomes entirely reasonable to hire an in-house engineer whose sole job is to find cost savings opportunities. Literally a “find unused stuff and turn it off” engineer.

I've spent some time doing this. There's always old systems people don't really understand, ownership is poorly defined, and no one knows what happens if you turn it off. It's archeology. Understand what the system is doing and how it interacts with other systems and the business. If it looks unneeded back it up, stop the VM, wait and watch for fallout, and eventually terminate it.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#113

Post by philsnow »

EvanAnderson wrote: post #20

I worked adjacent some telecom consultants in the 90s whose income was solely driven by a percentage of cost savings they could trim from telephone bills. Seemed like a very brash business model but they clearly knew there was gold to be mined. I keep thinking I should be doing "cloud optimization" work and being compensated this way. Slicing and dicing output from usage/billing APIs and providing an "optimized spend…

Like https://www.duckbillgroup.com/about/ ? There’s probably room in the market.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#114

Post by nodesocket »

I’ve mentioned this before, but probably one of the most egregious costs on AWS are NAT gateways and NAT bandwidth pricing. Typically I deploy one NAT gateway per AZ so looking at $99 a month just for three NAT gateway instances with zero traffic.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#115

Post by nlawalker »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

This topic got a lot of discussion in "I accidentally saved my company half a million dollars": https://news.ycombinator.com/item?id=38069710

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#116

Post by teaearlgraycold »

throwup238 wrote: post #26
pjc50 wrote: post #24

Earlier quoted context omitted.

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

We should be getting kickbacks from Amazon for all the work we’ve done for their bottom line.

I assume all of the unused resources end up subsidizing the rest of them.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#117

Post by Chico75 »

ponector wrote: post #47
zikduruqe wrote: post #2

Shit... I reduced our spend on one AWS account from $270K a month to $75K a month.

Great! Shareholders will be happy. Did you receive any bonus?

The problem is always around abuse. If it becomes known that you can get a big bonus by wasting a lot of money on useless infra first and then reducing it, other people will start playing the game.

How do you reward cloud cost awareness without creating perverse incentives?

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#118

Post by partiallypro »

I saved my previous company $4000-5000/mo on AWS billing just by auditing the AWS account and turning off unused machines that that old devs has spun up and deleting hard drives after backing them up to S3, just in case. No one had really even bothered looking at it for years and I did it in my "free time" at work without being tasked with it.

Ironically, I asked for a raise a year later and was denied, despite single handedly saving the company nearly $50000/yr. The raise I asked for wasn't close the cost savings I had brought. I left the company shortly after.

I saw someone else have a similar experience here and a comment to it was saying rewarding this produces a bad incentive...well, honestly why would I have even bothered cutting costs if I felt I wouldn't be rewarded? Not rewarding it just makes me half regret doing it at all.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#119

Post by myaccountonhn »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

What I've observed is that people don't really keep track of what they are spending. I like to set up weekly newsletters that show costs and also if there has been a decrease or increase. In bigger corps, you also should have team based tagging of resources so that specific teams get exactly what they are spending. At the very least, managers will look each week and be like "why did costs increase this week? What's going on?" even if the engineers don't care. "What's get measured gets managed" as they say.
Post reply on HN