Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cutting down AWS cost by $150k per year simply by shutting things off

tuananh.net

Post reply on HN
Print view View original

51–60 of 194 posts

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#51

Post by pyeri »

pjc50 wrote: post #24
darth_avocado wrote: post #16

It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…

The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears. And in many cases this is…

That's the whole ploy with Agile, isn't it? In the classical SDLC or Waterfall paradigm, everything was pre-approved and signed off, not just the cost or billing but even the software design itself. Any change in the process and the designers had to raise a change request. Agile changed all of that and now we know how bad things can get with that.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#52

Post by kevin_nisbet »

I've for a long time set my cloud VMs to shutdown on idle. I usually use it to also justify running a much larger VM to cut down on build and test times.

Just set a cron to run the shutdown command with a grace period. And then if you're working late, you just cancel the shutdown and the shutdown will be retried in a couple of hours. And have a script or command to just run the cloud API calls to boot the VM in the morning / when needed, and the environment boots in a minute or two.

For other stuff I've been tempted to do a more complicated setup, with something like a micro-vm as a proxy, that will do the shutdown / activation on TCP connection, but haven't gotten around to it.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#53

Post by bdcravens »

danfritz wrote: post #36

Cut costs down to 90% by going serverless and run everything on lambdas? Nothing keeps humming if it's not being used

Sure, after you reengineer your application. Even then, "serverless" apps often use persistent resources like databases, and your developers will likely spin up those resources for the same reasons as indicated in the article.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#54

Post by from-nibly »

Reacting to events to install security defaults (or any kind of defaults) sounds really error prone. Are people running AWS where devs just click buttons in aws and spin up random stuff? I thought we all decided that was dumb and switched to gitops/terraform?

Did I miss a new trend or something?

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#55

Post by happymellon »

darth_avocado wrote: post #50
happymellon wrote: post #41

Earlier quoted context omitted.

I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…

Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.

We had a battle to get AWS console access in the first place, after that I had to deal with:

> You need to request access every 60 days.

Luckily it's now added to my permanent role, but even then no billing access? FFS.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#56

Post by tuananh »

JohnMakin wrote: post #28

Looking forward to the kubernetes one - Most kubernetes clusters are designed for high availability, not necessarily for being able to quickly spin up/down and there’s often a lot of hidden complexity there (at least on aws).

hint: it's going to use the same platform. allow us having the ability to inject certain manifest into any eks cluster within the org.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#57

Post by bdcravens »

EvanAnderson wrote: post #20

I worked adjacent some telecom consultants in the 90s whose income was solely driven by a percentage of cost savings they could trim from telephone bills. Seemed like a very brash business model but they clearly knew there was gold to be mined. I keep thinking I should be doing "cloud optimization" work and being compensated this way. Slicing and dicing output from usage/billing APIs and providing an "optimized spend…

My employer has the same business model (we audit Fedex and UPS invoices for late deliveries, bogus surcharges, etc)

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#58

Post by joshstrange »

One small issue I have as a developer who can spin up just about anything on AWS is this:

I have zero insight into the costs.

Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard.

And before "well can look up what a t2.2xlarge costs and calculate it", sure. In a very contrived example I might be able to see what it costs but so many things are hidden/hard to see in AWS. For example, I recently spun up an RDS customer on my own AWS account. After testing for a while I decided it wasn't what I wanted and I deleted the cluster. Fast forward a month and my bill is well over what I expected (Like $30, no it's not a ton of money but it's my personal account and I wasn't expecting that charge). Come to find out it created a VPC as part of the RDS cluster (I think maybe it was for the RDS proxy? Still not sure) that didn't get deleted. I had to go chase that down and even that process wasn't easy. I had to make sure that it wasn't be used by anything else and then delete other things that were created when I made the RDS cluster before I could remove the VPC.

I was only able to do the above because I had access to the billing info. I would have left that VPC indefinitely on my work's AWS account by accident and been none the wiser.

I'm more than happy to take costs into account but without access to what things are actually costing us I can't help that much. Mostly because I need to know the costs to know what's worth optimizing. Sure I know I could improve X feature but if that costs us pennies a day (or month sometimes) then it's not worth it. Similarly if I know feature/infra Y is costing $XX,000/mo then I know I should rethink or investigate if that's correct/worth it.

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#59

Post by tuananh »

joshstrange wrote: post #58

One small issue I have as a developer who can spin up just about anything on AWS is this: I have zero insight into the costs. Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard. And before "well can look up what a t2.2xlarge costs and calcula…

billing transparency is very important.

in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill

Re: Cutting down AWS cost by $150k per year simply by shutting things off

#60

Post by bob1029 »

danfritz wrote: post #36

Cut costs down to 90% by going serverless and run everything on lambdas? Nothing keeps humming if it's not being used

Cost savings can be incredible if you use the FaaS product in the most aggressive way possible. For us, this means using functions as a simple translation layer between SSR web forms served directly as text/html and whatever SQL provider (ideally also on a consumption-based tier).

90% sounds just about right. We are seeing figures going from $120/m for a VM-based QA environment to $10/m for a consumption-based / serverless stack.

Post reply on HN