Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

231–240 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#231

Post by steren »

quesera wrote: post #178
steren wrote: post #107

Earlier quoted context omitted.

Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.

> We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it. That seems absurd to me. I think it makes much more sense to put the onus on the sophisticated customer to increase their maxScale to an unusual value. Users who "expect infinite scaling...and start with a load test" ar…

My response at https://news.ycombinator.com/item?id=25379846

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#232

Post by jontro »

heavyset_go wrote: post #212
steren wrote: post #107

Earlier quoted context omitted.

Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.

Given that this is a common problem, and one that can bankrupt individuals or their businesses, when is AWS going to implement spending caps that are easy to set up for new developers or business owners?

Still the really expensive thing here was the datastore reads. Cpu time was only 10% of the bill

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#233

Post by patrec »

beoberha wrote: post #179
ctvo wrote: post #146

Earlier quoted context omitted.

> They want to suck the maximum money from consumers before they realize. This is a naive understanding of how corporations like Google and Amazon work. Bad will and using gym membership tactics aren't how they scale or make money. Getting you to confidently try things knowing you won't get charged (the reason they have those free tiers) so you'll get your company, your start-up, your next side project on it is much…

I work in this space and you’re absolutely correct. Your last paragraph hits the nail on the head for pretty much every complain people have about the public clouds.

Right, so let's say Congress passes a bill that requires cloud providers to enable hard spending limits by start of February 2021, and eat any extra usage costs that exceeded a set limit.

What is your educated guess by when this feature would be essentially correctly implemented in AWS and GCP (essentially = negligible costs to the providers due to either false negatives (bills they eat) and false positives (PR fallout, when SomeSite gets shutdown despite not being over limit)?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#234

Post by mike_ivanov »

zigzag312 wrote: post #77
LeonM wrote: post #11

Earlier quoted context omitted.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Why full-time? It's possible to outsource IT administration to a local IT company and pay only for set-up and maintenance that is needed. Way less than 72k a year for many use cases. Also, companies that employ bunch of developers can find a developer that has IT administration expertise and allocate some of his time to this. Still cheaper than 72k a year or employing someone full-time, if IT requirements don't call…

Exactly. Or just buy a managed dedicated server - it's more expensive, but still it's a fraction of the full time sysadmin cost, and much cheaper than AWS.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#235

Post by asciimike »

Every time I see another post like this I always wonder how many people would be willing to buy "cloud insurance" where a premium would cover overages due to your mistakes in dev, outages, etc.

I don't have an exact billing model worked out, and I assume the insurance provider would mandate certain practices (e.g. setting up billing alerts that go to them, allowing them to view/manage infra), but curious if people here would be willing to pay for such a thing.

(My assumption is that most people who fall into this are too small to be willing to pay a reasonable % of their infra spend or change their infra practices to prevent this, but I'd be curious if this is something CIOs of companies who are thinking of "moving to the cloud but are leery due to cost concerns").

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#236

Post by meekrohprocess »

asciimike wrote: post #235

Every time I see another post like this I always wonder how many people would be willing to buy "cloud insurance" where a premium would cover overages due to your mistakes in dev, outages, etc. I don't have an exact billing model worked out, and I assume the insurance provider would mandate certain practices (e.g. setting up billing alerts that go to them, allowing them to view/manage infra), but curious if people he…

You might be able to get an insurance company to sign on to that sort of thing; have you shopped for quotes?

They'd probably want to look at a representative sample of "cloud overrun" events and see how much they cost, how likely they are, what reasonable measures could help prevent them, etc. But they aren't strangers to taking unusual bets.

Apparently a lot of big moonshot / X-prize type competitions fund their prize money through insurance. You present your research stating why the problem is hard, the insurance company works out how likely it is for someone to manage the challenge, and they come up with a quote for footing the bill in the unlikely event that someone wins.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#237

Post by sjg007 »

asciimike wrote: post #235

Every time I see another post like this I always wonder how many people would be willing to buy "cloud insurance" where a premium would cover overages due to your mistakes in dev, outages, etc. I don't have an exact billing model worked out, and I assume the insurance provider would mandate certain practices (e.g. setting up billing alerts that go to them, allowing them to view/manage infra), but curious if people he…

Does AWS implement spending limits? I would think that would be sufficient, if not, at least a start.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#238

Post by oblio »

steren wrote: post #229

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and pr…

Well, the real question for all cloud providers, for which I expect crickets as an answer, is:

Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#239

Post by donavanm »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource.

On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the customers users. And with billing alerts the customer is able to make the choice of whether the cost is worth it as it happens.

Note: Principal at AWS. Have worked to issue many credits/bill amendments, but dont work in that area nor do I speak for AWS.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#240

Post by joshmanders »

tunesmith wrote: post #224

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

I'm absolutely astounded that cloud providers allow stuff to get this way and don't even go "ope, this looks out of the ordinary, we should look into it" Nor do they offer the ability to straight up kill all services if exceeding a certain price set by the customer.

Cloud is still good though, I believe it's the future. I just don't believe in deceiving your customers to hopefully rack up a high bill with them.

I view not having the ability to say "Shut everything down if I go over $100/mo" the same as pre-checked hidden cross sales that MasterCard/Visa cracked down on in the adult industry few years ago. Just money grabs.

I will definitely be putting such measures in my cloud platform.

Post reply on HN