Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

341–350 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#341

Post by advisedwang »

oblio wrote: post #238
steren wrote: post #229

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and pr…

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

This is a outage waiting to happen for every customer:

- Early dev sets a limit.

- Product launches.

- Slowly grows.

- One day suddenly the entire business grinds to a halt. Globally. Across the carefully isolated shards. Everyone scrambles to figure out why! Tens of thousands of dollars are lost because of going $10 over a budget. End-users are lost. Trust is burned. If it's providing a critical system, maybe even people are hurt.

- Google then has to explain why they built in instant, global failure mode.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#343

Post by tunesmith »

sudcha wrote: post #332
tunesmith wrote: post #303

Earlier quoted context omitted.

You might want to take another look at your paragraph: > Having been a Googler for ~6.5 years and written dozens of project documents, incident reports, and what not, I knew how to put the case for Google team when they would come back to work in 2 days. That certainly reads as an advantage that most non-Googlers would not have.

Thanks for sharing, I see now what you mean. I'll share the doc I prepared and sent to Google in my consult, in one of the next posts.

FWIW, I didn't intend "deeply dissatisfying" as criticism against your writing, although I phrased that poorly so I can understand it coming across that way. If anything I feel for you when that unfair surprise hit you. It just sucks that it's possible, and that the odds feel against us when we're seeking a refund.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#344

Post by onion2k »

jasonpeacock wrote: post #102
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

You've explained why it's hard for Google to not give me resources I can't pay for, but that's not what I care about, or what I'm asking for. What I'm asking for is a feature where I set a hard limit of $100 and that's the most I get billed - if my account accidently uses $5000 of resources before Google reconciles the usage with my budget then Google automatically waives the additional $4900 and then limits my account in some way until the problem is rectified.

Practically every time these blog posts come up they end with the provider refunding the costs. I just want that refund to be a feature.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#345

Post by sandGorgon »

ggthrowaway2020 wrote: post #194
sandGorgon wrote: post #112

Earlier quoted context omitted.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution. In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying. We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, thi…

P.S. so im not a googler.

you misunderstand the intent of this - you basically set this. even if it fails (because messages are delayed), google will refund.

This has happened to us before - they do a refund - since you had set the limits correctly. In general, they are not super assholes. I actually dont know a case, where they have refused to refund.

AWS is better here - since GCP doesnt have a support dashboard. So the "chasing them" experience is much worse.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#346

Post by tilolebo »

The main reason why cloud providers can't provide this killswitch feature everybody talk about here is probably that they just can't.

I don't think any of them is capable of billing in real time.

So the monitoring would have to happen on resource consumption. This is already enforced with resource limits in many cases.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#347

Post by tpetry »

phkahler wrote: post #161
Paul-ish wrote: post #128

Earlier quoted context omitted.

The fact that the dashboards and alerts have a delay sounds like there might be difficult consistency stuff going on. Many nodes need to coordinate their usage and billing. It may be a difficult problem, but solving billing problems might not really motivate anyone at the company. It's not a "cool" problem for engineers and not profitable for product.

>> The fact that the dashboards and alerts have a delay sounds like there might be difficult consistency stuff going on. I think that's true. It's easier to measure usage and aggregate that data after the fact than to meter it in real time and stop at a limit. Those are very different things. What happens if you hit the cap while running multiple processes spread across a cloud? One improvement might be to throttle t…

Theres an easy solution: You set a limit and everytime a service needs to spend some many it allocates a small portion of the budget and after some threshold it will put unused money back to the budget. The only downside is that your spending limit will be reached to optimisticly, but i prefer that to paying thousands more than i wanted to. Knowing the system works like maybe a lower and higher threshold for the budget could be set.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#348

Post by spoopyskelly »

akh wrote: post #32

> Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144. Had we chosen concurrency of “1” request, we probably wouldn’t have even noticed the bill. > If you count the number of pages in GCP documentation, it’s probably more than pages in few novels. Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud service…

> Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work

Maybe you should brush up on that before launching another app that nobody needs?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#349

Post by oblio »

advisedwang wrote: post #341
oblio wrote: post #238

Earlier quoted context omitted.

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

This is a outage waiting to happen for every customer: - Early dev sets a limit. - Product launches. - Slowly grows. - One day suddenly the entire business grinds to a halt. Globally. Across the carefully isolated shards. Everyone scrambles to figure out why! Tens of thousands of dollars are lost because of going $10 over a budget. End-users are lost. Trust is burned. If it's providing a critical system, maybe even p…

They can put it behind a clear warning, do stuff like AWS does for bucket deletion (the bucket has to be empty, you have to check a box and manually type the full name of the bucket).

There are ways to design this, they can send notifications at 60% of the threshold, 80%, 90%, 95%. They can give you a grace period, put up prominent warnings in the console and for command line tools, etc. There are ways to do it, it's far from an intractable problem.

I'm not saying that it can't happen but do you want to bet that a certain percentage of their business, for all cloud providers, is from carelessness and resources still running when they shouldn't or using more than they expected? Especially for bigger companies where it's easy to miss something. Just like gym subscriptions or other kinds of subscriptions where they're banking on you not noticing for a long time ;-)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#350

Post by styren »

throwaway201103 wrote: post #168
that_guy_iain wrote: post #50

Earlier quoted context omitted.

> That sort of crap is the reason we host all our stuff on root servers. Having just started my own journey into building products for myself, pretty much the first thing I realised with my tech was I need to get dedicated servers instead of cloud just because it costs 100x less. > Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Exactly, if you really want kubernetes coolne…

> kubernetes ... is super easy to setup Can you point me to the super easy setup guide? Because I've tried a few and never gotten it working.

Have a look at https://github.com/hobby-kube/guide

I set up a 3 node cluster using it in an afternoon and haven't had any problems since.

Post reply on HN