211–220 of 397 posts
they know their systems aren't perfect, and if you velvet hammer them long enough, they'll do the right thing.
Earlier quoted context omitted.
A sane implementation would default this to a low value
Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.
Earlier quoted context omitted.
> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.
> I'm not sure that "colo is cheaper" is one. It absolutely is (cheaper, and a good argument). As an example: we're in the process of switching from Digital Ocean to Hetzner for a project, that will increase infrastructure performance (roughly memory/cpu/storage) by 4x and decrease costs by 4x. And no driving to the colo center is neccessary, as it's their dedicated server, so their on-site engineers do the hardware…
Earlier quoted context omitted.
It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
Once you get the alert that your budget is tripped you can go and see what's in storage via the console and delete it, only paying for a few hours of storage for things you don't want.
As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…
I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
Earlier quoted context omitted.
Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with…
You need separate costed quotas for each type of activity with a combined total at the bottom. You could also have a setting in the admin panel as to what the system should do: [ ] I want to keep going beyond my quotas (but email me) [ ] Please shutdown my site
The other issue is that many large customers pay different prices, so billing and quota aren't really tied to each other, and it wouldn't be easy to reconcile this.
As for the button... having been on the product side of building this button, there is no right answer: people will say they never got the email (or it went to the wrong inbox, or their dog ate their phone...) or that they never checked the box to "shut down the site" ("I didn't think it would do X that made my app not work").
Earlier quoted context omitted.
But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time. When there's no budget left, what do you do with those accruing costs for existing storage?
Moreover, once API calls are locked, what next? You can't delete files, and even if you can delete them, you aren't able to retrieve them before deletion... If a platform allows you to do those actions, then it's rife for abuse, and at public cloud scale that ends up being a far, far bigger problem than the occasional blog post that ends up as a refund (because the other blog post is "I got free storage forever with…
So much of it is so unnecessary to begin with. You can do so much with a cheap VCS or two without thinking about lambdas or cloud functions or kubernetes or who knows what. But these days you'd be forgiven for thinking it's dark magic.
You're not going to run up a 5 digit bill in a day by starting up on a few $10 VPSs. And you'll probably have an architecture that fits in your head to boot.
Also: The article title should really be "Saved 72k and avoided bankruptcy by being an ex-Googler."
As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…
I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).
Now, think some of these quotas can still lead to some pretty crazy bills.. but that is the point of at least some of them....