Why doesn’t GCP provide a way to say “shut down all my services if my cap is reached”?
Google Cloud customer wakes up to $18,000 bill despite $7 budget
11–20 of 69 posts
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#12I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
If they can COUNT it and charge based on that, that means they can count it and react.
If I, not having their budget or engineers, can have pretty much instant Prometheus event reacting to metrics, surely it wouldn't be too hard for them to have triggers like this -- somehow their AI can automatically ban people based on something, can't they do something for the customers?
They can, just don't want to.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#13I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
It’s hard on AWS as well, but I agree. There’s just no incentive for the billing experience to be better.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#14Earlier quoted context omitted.
It's the same fundamental problem as view counters, something Google is famously good at solving. Eventually consistent solutions are well-understood, and wouldn't have these kinds of massive cost-overruns.
Depends on latency. 24 hour delays on an eventually consistent counter used for billing absolutely would cause this problem.
It seems hard to believe that a one-hour delay on such a counter is impossible to achieve, and one hour would reduce the risk from "catastrophic" to "serious problem" in most cases.
Also, if implementing a cap is a desired feature that justifies trade-offs to be made, then it is psosible to translate the budget cap (in terms of money) back into service-specific caps that are easier to keep consistent. Such as "autoscale this set of VMs" and "my budget cap is $1000/hour", with the VM type being priced at $10/hour, translated to "autoscale to at most 100 instances". That would need dev work (i.e. this feature being considered important) and would not respect the budget cap in a cross-service way automatically, but still it is another piece in the puzzle.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#15Earlier quoted context omitted.
It's the same fundamental problem as view counters, something Google is famously good at solving. Eventually consistent solutions are well-understood, and wouldn't have these kinds of massive cost-overruns.
Depends on latency. 24 hour delays on an eventually consistent counter used for billing absolutely would cause this problem.
Yeah, there's an implicit assumption was reasonability.
But a big part of the value in large clouds like GCP is the network's interconnectedness. Plus even if there was some global event that made communications impossible only for the billing service, I'd still expect charges to top out roughly proportional to the number of partitions as they each independently exceed the threshold. GCP only has 120ish zones.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#16Earlier quoted context omitted.
I am the last person to defend Google Cloud and it's awful UX. With that said, when you go to set a budget it warns you "Setting a budget does not cap resource or API consumption. Learn more." with a hyperlink to https://docs.cloud.google.com/billing/docs/how-to/budgets?_g...
Click here to let the puppy life* * By clicking here you agree to kill it And you're defending that?
No, I think Google should provide easy tools to actually cap spending, instead of recommending you set quota limits on your APIs.
The article, and the comment I was replying to, make it seem like an error in the Google Budget system. I'm simply trying to say this system is working as designed and documented.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#17Earlier quoted context omitted.
I am the last person to defend Google Cloud and it's awful UX. With that said, when you go to set a budget it warns you "Setting a budget does not cap resource or API consumption. Learn more." with a hyperlink to https://docs.cloud.google.com/billing/docs/how-to/budgets?_g...
The only way I can read that is 'setting a cap does nothing' but reading that tells me that it only turns on email notifications. Not any better really. It's simply not a cap. It's an alarm.
Agreed. I don't agree with the underlying design decisions either, but you literally set a "Budget Alert" on Google Cloud. It's designed to be an alarm, not a cap. I was just trying to point that out
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#18I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
In the article it states that this person had an account that would have been limited to $2000 in usage.
And the system automatically upgraded them to higher spending limits when they crossed the $1000 in usage costs.
They could definitely make that an opt-in feature.
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#19I think I read somewhere that calculating and limiting cloud usage costs is a really hard problem. But I feel that if Google were motivated to do it, they can do it. It's hard, not impossible. They just don't care to solve this particular problem.
I mean yes, look at Corey Quinn [1] for example. He has built an entire career out of the fact that cloud billing trips people up.
(Generally, tech seems to skate by on creating insanely complicated things, knowing that given enough pain, people will start blogging about their solutions, ie effectively outsourcing the cost and effort of doing something about it.)
Re: Google Cloud customer wakes up to $18,000 bill despite $7 budget
#20Why doesn’t GCP provide a way to say “shut down all my services if my cap is reached”?
They have a $18,000 reason for that.