Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

331–340 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#331

Post by betwixthewires »

KorematsuFred wrote: post #314
betwixthewires wrote: post #308

Earlier quoted context omitted.

In my opinion, and maybe I'm an absolutist about this, the fact that there aren't these guardrails is opportunistic and predatory. Agile, iterative design and testing will inevitably lead to failures, that's the whole point. Marketing a cloud service to developers who need scalable and changing access to computing during that process should take that into consideration.

I do not think the intentions here are to be opportunistic and predatory but inability to empathize with small developers. A large customer will very likely just pay off few hundred thousand dollars extra expenses. It is only individual developers who are at risk here and cloud operators do not have much interest in them.

I don't know about that. Large customers can almost always find the capital to run their own infrastructure and save on cost. That's not to say that there aren't big customers of these types of services or that certain business models make using them more attractive than maintaining infrastructure yourself, but I would guess that revenue from these sorts of services are largely built on appealing to smaller customers, so their needs would be taken into consideration. Not taking potential cost overruns into consideration to me seems a bit deliberate.

To me it looks very similar to the personal checking overdraft schemes banks were using up until a few years ago.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#332

Post by sudcha »

tunesmith wrote: post #303
sudcha wrote: post #267

Earlier quoted context omitted.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

You might want to take another look at your paragraph: > Having been a Googler for ~6.5 years and written dozens of project documents, incident reports, and what not, I knew how to put the case for Google team when they would come back to work in 2 days. That certainly reads as an advantage that most non-Googlers would not have.

Thanks for sharing, I see now what you mean.

I'll share the doc I prepared and sent to Google in my consult, in one of the next posts.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#333

Post by Aeolun »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

Yeah, had that when I just started using it and it happily kept scaling like crazy. $200 bill in one day.

I never used google again.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#334

Post by scrollaway »

thanksforthe42 wrote: post #257
scrollaway wrote: post #176

Earlier quoted context omitted.

Why does any of this matter?

Different expertise. I'm guessing you are neither?

Incorrect guess, and it still doesn't really change anything. You're just playing with words. It's no more useful than a full thread arguing about a misspelling; Just pure noise.

Software engineering could learn a lot from, say, civil engineering. It could also learn a lot from interface design and I'm sure even microbiologists and astronauts could teach us a lot. Engineering is not special.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#335

Post by speed_spread »

dralley wrote: post #258
donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

>It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. Making the worst case scenario no worse than traditional infrastructure.

Correct. That argument assumes that every penny spent autoscaling has a positive ROI.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#336

Post by Aeolun »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

I feel less troubled by this in AWS because they actually have functional customer service.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#337

Post by arp242 »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

> The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource.

Well, this is true, but this is also true of a lot of limits, like limits.conf. Sometimes you really want to spawn loads of processes or open many files, but a lot of the time you don't, so a barrier to limit the damage makes sense.

There is no one solution that will fit everyone: people should be able to choose: "scale to the max", "spend at most $100", etc. If my average bill is $100, then a limit of $500 would probably make sense, just as a proverbial seat belt. This should never be reached and prevents things going out of control (which is also the reason for limits.conf).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#338

Post by Uptrenda »

It seems like you could easily build a fintech app that gives you a virtual pre-paid credit card to use with these services and gets funded only up until your budget amount. That might be a safer way to work with services like this. Could have a separate card where ever you expect a problem might occur. That would also avoid one service being able to take down everything else you need to bill

Card and billing management? Probably a legit need tbh

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#339

Post by uoaei »

camjohnson26 wrote: post #260
uoaei wrote: post #162

Earlier quoted context omitted.

> It is baffling why cloud providers don't have that option. ...is it? If a lazy dev leaves their corporate account open and you can bill it for their negligence, protected by the contract you already signed, you earn a lot of money. From a purely business perspective, it is stupid(!) to provide a stopgap for that. Edit: to be clear I am not advocating one way or the other. But it is surprising that people are "baffl…

Google is around a trillion dollar company, your $75,000 is a completely immaterial amount to them. Not to mention it would be a one time payment that would drive away customers and lead to bad PR like this post.

So... where else are you going to go if you don't like these policies?

If everyone has this policy, Google, Amazon, Microsoft, and the rest are in a good place. And suddenly it's the "industry standard."

This hypothetical is already enacted today...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#340

Post by Retailer »

aidenn0 wrote: post #294
oblio wrote: post #238

Earlier quoted context omitted.

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

My guess is that the billing logic is separate from the application logic. There's probably a delay between the two and mostly one-way communication.

1) But they supported this before on GAE. GAE had 'spending limits'.

2) Also if they are able to figure out when you've hit your daily free quota and cut you off almost immediately, how are they not able to figure this out?

Post reply on HN