happened to me something very similar, I was using cloud run to fetch some subreddit posts and ended with a "recursive" way, because of that, billions of invocations was made... luckly, I was at the front of the computer and stopped before, but the bill was around $4000! I contacted the google support and explained all, they "forgot" my debit because of the bug
Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
91–100 of 397 posts
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#92This is most developers' worst nightmare when it comes to a completely new environment generally, and Cloud solutions specifically.
It's easy and pointless to say they should have done things differently. Worse than pointless. Obviously they should have, and kudos to them about being open about the compounded mistakes.
Still, this strikes at the fears that lie in the heart of any reasonable, honest developer doing something completely new.
New developers should be cautious about cloud platforms, but they were! Not cautious enough, obviously, but they did set limits they thought would be honored.
Platforms should have hard monetary limits at the account level, clearly, as well as an option to turn them off. Shame on all of them which don't.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#93Earlier quoted context omitted.
You forget your own cost here. A full-time system administrator costs more than 72k a year.
Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year
Jesus no wonder many IT professionals in europe want to come here.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#94I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#95> I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000. Super stressed, and not sure what happened, I clicked around, trying to figure out what was happening. I also started thinking of what may have happened, and how we could “possibly” pay the $5K bill. > The problem was, every minute the bill kept going up. > After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wa…
I think the article said the Firebase dashboard data was at least 24 hours behind what they were getting billed
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#96The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.
This is my worst nightmare. Lol. I guess now is a great time to give Azure a shoutout for sitting on their hands for 8 years without so much as a response to the community for half a decade [1]. At least AWS allows using a prepaid credit card so they’ll need to call me if things go haywire. I bet if that $72k charge went through it would have been much harder to get out of. “Sorry, we don’t have the money” is a much…
> “Sorry, we don’t have the money” is a much better negotiating position than “can we please have our money back?”
I agree but why would you like to be in either position anyway? The so-called cloud services are terribly overpriced when compared to traditional servers.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#97It's "fantastic" how Google by the end of the article still come out as a good and friendly bunch...
Ridiculous ... it's designed to charge you, upgrade you and makes u spend as much money as possible. And 24 hours after it happend they show it to you in the dashboard.
Great.
Just use dedicated servers for the start! It will hurt way less and you can easily upgrade to the cloud later, IF necessary.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#98Meh. I see this all the time with developers who want to abstract everything away and not worry about the impact their poorly performing code is having on the infrastructure or on the $$bottom line. Time out? No problem, we will just spin up more instances. I have heard that so many times. Maybe your code is just bad.
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#99You can ask for a good faith billing adjustment. GCE or AWS is well aware that things happen, and collecting something is better than collecting nothing.
Part two of the article indicates they did that, and had the bill nulled out.
from part 2:
"After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions Google let go of our bill as a one time gesture!"
Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt
#100Earlier quoted context omitted.
It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.
They want to suck the maximum money from consumers before they realize. I have very little money so I just don't use their services because a mistake would be disastrous. They might be losing out on me making a unicorn app on their platform. It's unlikely, but while the possibility of catastrophe exists I'll stick to not using them. That extends to not recommending anyone uses them either in case the worst happens.
I have very little money...
Then the harsh reality is: companies don't care. Yeah, your app might turn out to be a unicorn, but the overwhelming odds are that it won't. And no one cares that you'll tell your other broke friends to avoid the service.
We'd all like to think it to be different, that a company might care about appeasing my broke ass. But as already pointed out, they want the whales. I also wonder, despite the number of years "cloud services" have been around, if companies aren't still trying to figure out a gazillion other things and limiting customer spend might be a bit low on the priority list.