Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

101–110 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#101

Post by mr_00ff00 »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

Would be a random coincidence, but if you work at a large bank, I think I may know the team that had that 300k test db lol.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#102

Post by sotix »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

I consistently tried to push for cost management at my last job, but the product manager just wanted to push new features he could show off to management above him. We let costs inflate to ridiculous levels despite my constant discussion around the topic. Software engineers ultimately had no say in the matter.

I was laid off this week in a mass layoff because the company doesn’t have enough money to pay all of us anymore. It’s disappointing to see, and I wonder how many other teams ignored these optimizations and how much unnecessary total cost it all summed to.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#103

Post by tehlike »

cbg0 wrote: post #87
tehlike wrote: post #32

Cutting down AWS cost by 90% by simply moving to hetzner.

If you're running a small setup and don't need any value add products or multi-AZ/multi-region this might work, but Hetzner and major cloud providers are by no means comparable. Hetzner offers a 99.9% uptime guarantee only on their network. AWS has SLAs for every product offering - EC2 for example starts paying out credits if they fall below 99.99% uptime. If you're a user of various managed cloud products, these wil…

Managed is good, but opensource is decent these days.

You need postgres? Use crunchydata postgres operator or cloudnativepg. Need multiple regions? setup wireguard.

IT's more work, but might not be a lot of work.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#104

Post by pizzafeelsright »

StratusBen wrote: post #89

Disclosure: I'm CEO of https://www.vantage.sh/ -- a cloud cost observability platform. I previously worked at both AWS and DigitalOcean. For people looking for how to save money on AWS - I'd [selfishly] recommend connecting up to Vantage. We profile AWS for all sorts of savings and give you the information on how much we can save prior to you paying us. It can be a good gut-check if nothing else on how well optimized…

Silly request. The amount of paperwork, lawyers, and time required to connect our service makes it impossible to validate savings.

Is there an offline method? I have not looked at vantage to see if it's possible.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#105

Post by 8organicbits »

You should always calculate if you're actually going to see cost savings. Counterintuitively, running for fewer hours can increase your bill if it causes you to switch to on-demand pricing [1]. There's a break even point you need to get past.

[1] https://alexsci.com/blog/modeling-on-demand-pricing/

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#106

Post by pid-1 »

ary wrote: post #72

> The best optimization is simply shutting things off This is the way. A similar idea has been bouncing around in my mind for a while now. An ideal, turnkey system would do the following: - Execute via Lambda (serverless). - Support automated startup and shutdown of various AWS resources on a schedule influenced by specially formatted tags. - Enable resources to be brought back up out of schedule when demand dictates…

> Operate as a TCP/HTTP proxy that can delay clients so that a given service can be started when it is dormant or, even better, the service isn't serverless but you want it to be. This can't work for everything, but perhaps enough things such that the need to run always on services is reduced.

My team mostly builds internal stuff and we save tons of $$$ by using Knative + Karpenter, which basically does that on container + EC2 levels.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#107

Post by happymellon »

macNchz wrote: post #90
happymellon wrote: post #41

Earlier quoted context omitted.

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…

> getting as much as possible out of a system for cheap is great fun In certain circumstances, absolutely, however it's extremely aggravating to be in the position of being constantly pestered to ship features faster without the authority to overprovision some of the infrastructure the software runs on. Waking up in the middle of the night because we saved money by allocating too little disk for the primary database…

Completely agree.

I didn't mean that engineers love having no control over their systems, I just see the labours of love that get posted here about getting nginx to throw out 1000 pages a second on an Atari 800, or getting LLMs designed for $2000 GPUs running on a phone.

The question should be, we currently cost $X a month and we need to half it because [reason], what can we do to bring it down? Which might be reducing hardware, or maybe something else, might be both. Puzzles can be fun.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#108

Post by bdcravens »

tehlike wrote: post #100
bdcravens wrote: post #64

Earlier quoted context omitted.

Was doing some research this weekend on cloud exit. Hetzner is attractive, but our company is pretty much limited to the US (no international companies due to our current business model). How practical would it be? Also, I've seen a lot of concern over blocked IPs, especially for lower-cost hosts. Is that an issue with Hetzner?

Hetzner cloud supports us-west and us-east.

I was looking at dedicated hardware. If I decide to stay in anyone's cloud, I'd stick with AWS.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#109

Post by ExoticPearTree »

darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

In the organization that I work in, costs are transparent to everyone involved and most people are aware of the need to keep costs as low as possible.

One of the downsides with this approach is that engineers/developers are not very good business people and don't really understand the notion of "the cost of doing business". And from time to time we have issues with "but it costs $70 more per month", and spend $1000 to optimize those $70 :)

In the end, even with some of the wrinkles mention above it helps and saves money when costs are transparent and readily available for anyone.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#110

Post by belter »

joshstrange wrote: post #58

One small issue I have as a developer who can spin up just about anything on AWS is this: I have zero insight into the costs. Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard. And before "well can look up what a t2.2xlarge costs and calcula…

You are aware of this? - https://calculator.aws/
Post reply on HN