Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

241–250 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#241

Post by tpmx »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

The upside for the noob trying out/learning is huge.

I'm no longer that person, but I think GCP/AWS are just being lazy about this - perhaps because they earn a lot of money from engineer mistakes. Of course it's possible to create an actual limit. There'll be some engineering cost, like 0.5%-1% extra per service?

Edit: Being European I think legislation might be the fix, since both Amazon and Google have demonstrated an unwillingness to fix this issue, for a very long time.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#242

Post by dijit »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit.

Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code):

https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#243

Post by jcelerier »

sofixa wrote: post #105
jcelerier wrote: post #53

Earlier quoted context omitted.

35k is the gross which encompasses a big part of that except office space.

No it's not. In France companies pay ~1.5-2x gross salary in total, gross going to the employee ( some of it getting deducted by the state), rest going for health insurance, taxes, etc.

If you check the actual numbers here - https://entreprise.pole-emploi.fr/cout-salarie/ you'll see that for e.g. 30k€ gross yearly, a company shells out 41k, which is definitely not 1.5-2x total and still very far from the 75k mentioned above

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#244

Post by joshmanders »

LordHeini wrote: post #42

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

> Even trying to read the amazon pricing for their instances, hours and what not, drives me insane.

I get your sentiment but the pricing is that way because they want to charge you exactly what you use, not for reserving stuff.

For example if you deploy a EC2 instance that comes out to be $15/mo total, and you deploy it on say the 10th of the month, do you want to be charged the whole $15? No, you want pro-rated.. But what if you only need that instance for like 6 days? Then what? You gonna do the math to figure out what it would have cost you yourself, or just read it per hour billing?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#245

Post by shaicoleman »

oblio wrote: post #238
steren wrote: post #229

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and pr…

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

AWS recently released AWS Budget Actions which should allow you to do that.

https://aws.amazon.com/blogs/aws-cost-management/get-started...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#246

Post by Alupis »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

Why not alert thresholds, configurable by the user?

Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire.

Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechanisms and the user chose to ignore them.

Additionally, Google et al should protect _themselves_ by tracking usage patterns, and reaching out to customers that grossly surpass the average billable amount - just like OP with their near $100k bill in 1 day. Zero vetting to even have a reasonable guarantee the individual or company is even capable of paying such a large bill.

And then what? Sue a company that doesn't have $100k for $100k? This makes zero sense.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#248

Post by paxys »

cogman10 wrote: post #206
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

This is already present and very easy to set up.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#249

Post by patch_cable »

Alupis wrote: post #246
donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechani…

Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#250

Post by alasdair_ »

dijit wrote: post #242
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

The article says that they had a limit in place but that in practice the billing limit lags up to 24 hours behind the "real" number.
Post reply on HN