We do this. It was a bit faffy to set up, but having dev and staging shut down at night and only be started as needed that day saved us a fortune.
Cutting down AWS cost by $150k per year simply by shutting things off
121–130 of 194 posts
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#122One small issue I have as a developer who can spin up just about anything on AWS is this: I have zero insight into the costs. Yes, my company could turn that on for me but it's rare that they do so it's nearly impossible to know if I did something that costs a lot of money (relatively or in general) without access to the cost explorer/billing dashboard. And before "well can look up what a t2.2xlarge costs and calcula…
You are aware of this? - https://calculator.aws/
I’m very aware of that tool but it’s far from perfect. I’ve spec’d things out on that then seen very different prices when I actually create things in AWS. In part because the tool doesn’t take some things into account or because sometimes it’s impossible to guess your usage for a new feature.
I don’t believe the VPC was factored in when I used that calculator, even after selecting RDS Proxy.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#123Earlier quoted context omitted.
Hetzner cloud supports us-west and us-east.
I was looking at dedicated hardware. If I decide to stay in anyone's cloud, I'd stick with AWS.
Yeah Hetzner doesn't have dedicated in the US, and not sure if it will in near/mid future.
Still, Hetzner cloud is pretty good option, and there's more support coming on building on Hetzner.
https://www.ubicloud.com/ (from founders of citus) is mainly/currently targeting hetzner, for example.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#124Earlier quoted context omitted.
I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management
Which is why your slice of the organization gets a cloud budget. Don't keep your budget under control, well no bonuses for the employees. >Show me the incentive and I will show you the outcome
Look at Mr bonuse-pants here. Y'all get bonus for writing software?
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#125I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the savings for myself as a reward, the answer would be a very clear "no" to both. So overall, as an end developer I really have no incentive to work harder and ensure lower operating costs for my company, and I'm sure most developers in the industry are in the same position as me.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#126I've mentioned this before, in my company (big media company) I saw some S3 costs creeping up each month. I looked into it and it was a system we abandoned that was still copying files to this bucket. I reached out to the team and they turned it off, it saved us $1m a year. The higher-ups rewarded me by telling me that a team should have caught this so I should meet with them now.
It's truly fascinating how companies won't bat at an eye at spending ungodly amounts of money on things they don't need, but will sweat profusely at the thought of a tiny fraction of that going towards additional compensation.
Something something snakes, something something unintended consequences.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#127I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…
Part of it is that it creates an incentive to create wasteful systems, only to "optimize" them later to rack up a bonus. Even if it gets changed to only pay out for reducing spend incurred by other engineers, it's possible to collude in such a way to extract bonuses from the company.
A better way to have aligned incentives for the company and the employees would be to allocate a bonus pool for the entire company, from which AWS expenses are taken out of, but that might be a bit unorthodox.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#128Earlier quoted context omitted.
Which is why your slice of the organization gets a cloud budget. Don't keep your budget under control, well no bonuses for the employees. >Show me the incentive and I will show you the outcome
But bonuses are not related to the cloud costs. Nobody in engineering cares about spending because there is no benefits on doing so. Even more: most people are on fixed salary and will get paycheck anyway no matter how low their effort is.
Hence
>>Show me the incentive and I will show you the outcome
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#129I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…
How are they going to be sure you're not just farming cobras[1]?
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#130I was thinking about this recently. I work at a large company with untold millions in AWS spend. I'm 100% confident that I could shave off a few thousands (maybe even tens or hundreds of thousands) from the bill with a little bit of effort on my side. If I go up the management chain and ask if (1) I can make this an official project and put it on the roadmap or (2) I can do this on my own time and keep some % of the…
This is one of many insights that hint at why biz-facing cloud architecture is so popular, wasteful, and profitable.
The incentives are designed to form an enormous cash siphon. From aggressively marketing toward fearful & liable (or maybe just tech-cost-illiterate) upper-management to the silencing effect that the low-rung experts experience when sounding the alarm.