I’m convinced that once a company reaches ~$10m/year in AWS spend it becomes entirely reasonable to hire an in-house engineer whose sole job is to find cost savings opportunities. Literally a “find unused stuff and turn it off” engineer.
Cutting down AWS cost by $150k per year simply by shutting things off
81–90 of 194 posts
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#82Earlier quoted context omitted.
hint: it's going to use the same platform. allow us having the ability to inject certain manifest into any eks cluster within the org.
Have you done this or attempted this yet? Every kubernetes cluster is different, but in my time working with them the last several years I anticipate the following issues: - dependent services not coming up in the order you expect/want - issues draining nodes due to crashlooping/erroring pods (can also be caused by dependent upstream services going down in wrong order) - Persistent Volume retention/synchronization -…
> I’ve had many managers over the years pitch this idea of “rapidly deployable/destructable EKS clusters” and the projects always get killed due to the complexity around this.
This is exactly what we do: blue green eks cluster.
We just thought if we do it on monthly basis, DRP will be piece of cake :)
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#83Earlier quoted context omitted.
I think "engineer" isn't really the correct word to use for the artisans who build much of the tooling used by most companies. An engineer either wears a striped hat and drives a train, or, went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people." It is expected that an engineer can predict with reasonable accu…
“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…
Engineers manage risk and cost. They certainly make mistakes, like those couple buildings that are famously leaning over in SF and NYC, or the citycorp center where they got the wind sheer loads wrong and had to hot patch the building.
But looking at the malarkey that goes on in "software engineering" or whatever -- clearly not engineering, at least not where I've seen it.
Engineering: a process of repeatably solving an understood problem predictably.
Craft: a process of solving an understood problem.
Science: a process of solving a problem without an exactly understood outcome.
Art: a process of working.
These are all made-up definitions.
I'd expect a software engineer to give me a system that locally caches and verifies distribution artifacts and validates changes -- a craftsperson who gives me a tool chain that yeets goo from the internet and builds on that without validation is not, in fact, an engineer. They could be quite practiced at the art of building working systems, but they're not managing risk....
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#84Re: Cutting down AWS cost by $150k per year simply by shutting things off
#85It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…
This is entirely artificial: I now work at a company where we know very clearly what our infrastructure costs. Yes, we know the exact costs (what was negotiated, not what is on the public pages).
And we celebrate costs slashing as much feature delivery and other stuff.
But this is entirely a management problem: at my previous job, only one manager (skip-level manager from my point of view) knew what exactly were we paying for infrastructure.
That moron wouldn't share that information with us engineers managing infrastructure of course, so there were a lot of infrastructure choices that didn't really made sense according to the public prices but (I guess?) made sense according to a price sheet we didn't know.
So we didn't know what we were spending, didn't have the basic data to estimate the price of a new solution or a new service and didn't have the data to determine how much would we be saving by making changes (optimizing stuff etc).
I fought that battle for a bit but then i just said "GFYS, i'm not going to have fights with you so that you can save money" and let go. Later i left the company completely.
Former colleagues tell me it's even worse now: there are consultants from the cloud provider involved, they know the pricing deals, and whenever the topic comes up the manager shushes the consultant so that the engineers don't hear the prices.
tl;dr: it's an entirely artificial problem, and it's most likely a cultural/management problem.
edit: and i'm not even talking about incentives, as somebody else has correctly pointed out.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#86Couple of years ago I saved about 14 mn of revenue per year for my company. I got a 250$ bonus for it.
Learned a lesson that you only have to "spend" (forgo) $250 to cause that company $14m in losses (that you could have prevented)
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#87Cutting down AWS cost by 90% by simply moving to hetzner.
If you're running a small setup and don't need any value add products or multi-AZ/multi-region this might work, but Hetzner and major cloud providers are by no means comparable.
Hetzner offers a 99.9% uptime guarantee only on their network. AWS has SLAs for every product offering - EC2 for example starts paying out credits if they fall below 99.99% uptime.
If you're a user of various managed cloud products, these will cost quite a bit to replicate on Hetzner and you'll be spending money on personnel to build these out and maintain them instead of just paying for the cloud product on AWS/GCP/Azure.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#88Earlier quoted context omitted.
“engineers stick to doing things that are well understood and predictable” I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to e…
Engineers manage risk and cost. They certainly make mistakes, like those couple buildings that are famously leaning over in SF and NYC, or the citycorp center where they got the wind sheer loads wrong and had to hot patch the building. But looking at the malarkey that goes on in "software engineering" or whatever -- clearly not engineering, at least not where I've seen it. Engineering: a process of repeatably solving…
What makes software engineering special is the systems are more complex and are cheaper to test and break. You get a completely different engineering culture when you can roll back a bad change after seeing it fail during the canary push. That, and what's usually on the line is money, not life. I'd feel a lot better making a $1M mistake than making a mistake that killed someone.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#89Disclosure: I'm CEO of https://www.vantage.sh/ -- a cloud cost observability platform. I previously worked at both AWS and DigitalOcean.
For people looking for how to save money on AWS - I'd [selfishly] recommend connecting up to Vantage. We profile AWS for all sorts of savings and give you the information on how much we can save prior to you paying us. It can be a good gut-check if nothing else on how well optimized you are.
Re: Cutting down AWS cost by $150k per year simply by shutting things off
#90It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies. I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were payi…
I found that the problem happens mostly when companies 1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun. 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in…
> getting as much as possible out of a system for cheap is great fun
In certain circumstances, absolutely, however it's extremely aggravating to be in the position of being constantly pestered to ship features faster without the authority to overprovision some of the infrastructure the software runs on.
Waking up in the middle of the night because we saved money by allocating too little disk for the primary database or because the latest release included new dependencies that increased memory usage and the OOMKiller is picking off web servers like a wolf in the lamb's pen, or we're just swapping our way to hell while web requests 502...eh. Not for me.
More visibility into costs, though, absolutely agreed. Engineers should know that when they turn on some new cool serverless gizmo and then forget about it, it's costing $ each month.