Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

41–50 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#41

Post by happymellon »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I found that the problem happens mostly when companies

1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun.

2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse?

It's rhetorical really as I know why. Terrible architecture from "enterprise". Stick everything in a single account so it's hard to figure out how much is your spend. All 3000 databases, and make sure your k8s cluster is 5 8XL boxes so no one can scale down excess capacity.

Classic I Burn Money consultancy!

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#42

Post by ponector »

datadrivenangel wrote: post #6

If you're spending $425k per year on non-production AWS resources, you have an interesting setup.

In one project we had testing setup which costs 600k USD allually. It was three times more expensive than production setup we had for product which was more than 3 year old. Nothing special, just mongo and Kafka with enormous size. If you run automation tests Manu times per day but do not clean anything - you'll get mongo with terabytes of test data. And then, on top there was elastic search which multiply bills.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#43

Post by righthand »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I had to implement a 2nd deploy for a QA environment, and my first question to the infrastructure team was “won’t this be costly is there a better way to handle this?” They shrugged off the cost and said they would optimize my deploy once I was done with the initial implementation. 6 months later their optimization was to undo all the work not because it wasn’t a good implementation but because it revealed how much non-optimization had went into the QA environment before I even touched it. A lot of cost is probably due to the “we just taped these two things together” strategy for lower environments.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#44

Post by racl101 »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I tend to have the opposite problem. I obsesses over the cost of things, and am pretty bashful about bringing it up to my manager, and he's always surprised that scaling some resources doesn't cost more. But I learned the hard way as a contractor about letting these resources run crazy and had to pay out of pocket so I have PTSD about it, which is why I'm vigilant.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#45

Post by bdcravens »

cosmotic wrote: post #38
ejs wrote: post #14

This is especially easy if you can shutdown environments that are only used for dev/staging tasks. With 168 hours in a week - how many hours do those things need to be running? I run a little tool for Heroku to make it easy to do this kinda thing.

Often the continous use discounts make regularly turning on and off a wash.

This assumes they have something like RI etc for those resources. Those are typically used for production, but far too often, dev/test resources are usually turned on ad hoc.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#46

Post by tuananh »

happymellon wrote: post #41
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…

> 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse?

This is so true. billing transparency is very important.

in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#48

Post by schnebbau »

anuptalwalkar wrote: post #17

Couple of years ago I saved about 14 mn of revenue per year for my company. I got a 250$ bonus for it.

This is why you should start the conversation with "I have drawn up a plan to save the company $14M per year. I will execute this plan in exchange for $7M upon completion."

If they say no then just go back to your regular duties.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#49

Post by dilyevsky »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

> It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis.

That’s because they were explicitly told not to worry about costs for the last 10 years so majority of ICs at this point never had to do it their entire careers

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#50

Post by darth_avocado »

happymellon wrote: post #41
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…

Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.
Post reply on HN