Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who is accountable for cloud costs in your org?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 20 posts

Ask HN: Who is accountable for cloud costs in your org?

#1

Post by devashish86 »

Context: I lead the DevOps team in a mid size engineering org (60 engineers approximately). The product is a B2B SaaS product. The organization started taking cloud costs seriously early 2024 and my team worked throughout the year to make infrastructure changes to reduce our cloud costs. This included projects like removing manually created stale infrastructure, automating infra management, rightsizing, purchasing the right Savings Plans(and RIs), and many others. Now we're at a place where infra-only projects to optimize cloud costs are pretty much exhausted. On the other hand, most of the anomalies (and surprises) regarding cost spikes come from application level changes. This causes serious problems because not only these anomalies are identified late into the deployment lifecycle, but these anomalies are inherently harder to resolve quickly. An example of this is when a service triggered a downstream workflow which started spawning additional background jobs (10x in production) which blew up the cost projections.

So, my question to the group is - Who do you hold(or should hold) accountable when cloud costs spike unexpectedly: the engineers who write the code, the platform team who manages the infrastructure, or the product managers who set the requirements? (My current solution is a mix of platform team and engineers but we're still trying to formalize the accountability model.)

Re: Ask HN: Who is accountable for cloud costs in your org?

#2

Post by talonx »

Rather than focusing on accountability as the starting point, I would suggest building tooling and visibility so that cloud costs are visible across all layers including application and infra. Once you have this, accountability becomes easier.

Each application team should be able to view the total cost of running their service - and thus be held accountable to reduce costs when necessary.

Without data you are running blind. Cost optimization cannot be solved by a standalone team - it has to be owned by everyone.

Source: Personal experience reducing cloud costs in a slightly smaller team.

Re: Ask HN: Who is accountable for cloud costs in your org?

#3

Post by YuriNiyazov »

Visibility is essential, and visibility is owned by the devops team. ALl the resources have to be tagged with the team that owns those resources, and you should have a meeting once a month or so with that team where you bring to them a report of "here are all the resources we have assigned to you, do you agree?" and if they agree, then great, they are responsible for it, and if they don't agree, then now you have to have a conversation where you bring in the PMs and the EMs, and the head of finance if you need to.

And it's essential that you check in regularly, rather than just give them a dashboard and say "ok, you go look here and tell me if anything is amiss", because they will never look.

Re: Ask HN: Who is accountable for cloud costs in your org?

#6

Post by MattGaiser »

The problem with the "everyone" model being pitched here is that it may as well be a synonym for "nobody."

I've worked for a few orgs where quality and testing were "everyone's" responsibility and it ultimately led to everyone pushing it off their plates and lots of it simply not getting done. Why? We could collectively borrow against the future and "everyone" being responsible meant that nobody could be held accountable, as then the debate would be in deciding fractions of responsibility.

It also encouraged those with other incentives, like product, to lean heavily on that to ship more features over doing reliable tech work as they figured the debt would be someone else's problem down the road.

People have this naive idea that people who are given responsibility will step up. There are those that do, but the rest often see the far easier path of externalizing problems and frankly most jobs reward that as they don't see externalities well.

I would have it so that platform team is responsible for identifying and engineering is responsible for fixing it. I am not sure that either team would have the skills needed to prevent such things from happening, so perhaps canary deployments would be the way to go if it is a substantial risk in your domain.

Re: Ask HN: Who is accountable for cloud costs in your org?

#7

Post by MattGaiser »

re-thc wrote: post #4

> or should hold EVERYONE. Like with everything on Earth it's everyone's responsibility. It all adds up. The way where everything is in isolation and silos really doesn't work and with everyone not having the full picture nothing is ever optimal.

Everyone can't have the full picture as too much knowledge and information are required and even if they did, their incentives usually aren't aligned with caring.
Post reply on HN