Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

301–310 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#301

Post by xwdv »

salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

If I were in this situation I would probably offer 10% of the bill to the employee as a reward for their help.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#302

Post by xwdv »

sudcha wrote: post #182
Alupis wrote: post #165

As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat t…

OP here. Bankruptcy fear was real at the time. Google has at least a few thousand lawyers on payroll. They probably also have a process of handling delinquencies and sending them notices. A quick look at the lawyer fee to just manage the case, let alone fight it, is enough for bootstrapped company to raise hands. +1 to bad actors possibility. I shared this with Google team, I'm not sure what they have done since. We…

This is why you create a shell company to use cloud services with while your real company leases the servers from that company. As soon as you run up a bill you can’t pay you shut down the whole shell company and reopen a new one.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#303

Post by tunesmith »

sudcha wrote: post #267
tunesmith wrote: post #224

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

You might want to take another look at your paragraph:

> Having been a Googler for ~6.5 years and written dozens of project documents, incident reports, and what not, I knew how to put the case for Google team when they would come back to work in 2 days.

That certainly reads as an advantage that most non-Googlers would not have.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#304

Post by Retailer »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

If you're on App Engine, we did an article about that

https://retailingplatform.com/blog/how-to-control-costs-on-a...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#305

Post by CamelCaseName »

dsymonds wrote: post #287
Exuma wrote: post #270

Earlier quoted context omitted.

You only wrote half the story and no technical details...... why even write a post at all, it's just clickbait and you give no other information how what went wrong, how it got fixed, what exact technical problem was...

Did you read Part 2, linked from the bottom? It has plenty of technical details.

Up until this comment, I did not realize that was a link either.

Perhaps OP can edit the bottom to make it more clear?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#306

Post by Retailer »

Alupis wrote: post #246
donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechani…

Google has alert thresholds (you set it up under your Budget). But practically speaking, an alert is not enough - what if you are unavailable to get the alert, it comes in the middle of the night, etc?

A better solution would have been 'limits' which they used to have (at least for Google App Engine) but which has been deprecated.

We had to spend sometime to research and see if there was a work around because just like the author of the article, we were quite worried about suddenly consuming a huge amount of resources, getting a spike in our bill and our accounts being cut off/suspended because we hadn't paid the bill. We've documented our solution here

https://retailingplatform.com/blog/how-to-control-costs-on-a...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#307

Post by Retailer »

asciimike wrote: post #191
NegativeLatency wrote: post #140

Earlier quoted context omitted.

Probably hurt revenue ;)

As a former App Engine PM who spent a lot of time with billing/quotas (though, not the one who deprecated this feature), it's likely due to some combination of: - hard limits caused downtime more often than they prevent these blog posts - hard limits were inconsistently enforced, even within GAE - platform wide quota notifications were implemented (reached "GA"), leaving the question of "how a developer wants to hand…

Personally, I don't think this is a good enough reason. Worst case, if I experience an unplanned shut down, I will increase my spending limit. Removing the feature entirely because of this just doesn't make sense.

When I also think of the fact that Google tied it to requiring a credit card for almost every single transaction even if it is free gives the impression that it is for financial purposes (aka a way to get more out of developers or those who might be free-loading on the free tier of App Engine)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#308

Post by betwixthewires »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

In my opinion, and maybe I'm an absolutist about this, the fact that there aren't these guardrails is opportunistic and predatory. Agile, iterative design and testing will inevitably lead to failures, that's the whole point. Marketing a cloud service to developers who need scalable and changing access to computing during that process should take that into consideration.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#309

Post by betwixthewires »

tunesmith wrote: post #224

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

I'm with you. I have yet to use any of these cloud computing providers for building or testing anything, and it is partly due to this (and partly due to privacy and confidentiality considerations).

"Yay! NBD! Google is the best! All I had to do was work there for a few years, rub elbows, make connections and ask for favors!" Really, google would be the best if there was no way to accidentally go over your stated budget cap by 86 million percent, or at the very least have a policy to refund people who can demonstrate that this is what happened.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#310

Post by vp8989 »

brundolf wrote: post #184
serial_dev wrote: post #23

Earlier quoted context omitted.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

For hobby projects you probably don't need auto-scaling, and should use a provider that charges a fixed monthly rate. You'll "waste" a little bit of money on unused uptime, but for a hobby project it will be a minuscule amount.

But then I don't get to be a "serverless hero" and write blog posts about how my side project (that no one uses) costs $0.000034 to run instead of $5.
Post reply on HN