Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

291–300 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#291

Post by donmcronald »

dathinab wrote: post #263
dijit wrote: post #242

Earlier quoted context omitted.

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

> Note: There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget…

> it's really hard to find a good setting which doesn't allow to much "over" cost and at the same time doesn't hinder the intended auto-scaling use-case

> I created a new GCP project ANC-AI Dev, set up $7 Cloud Billing budget, kept Firebase Project on the Free (Spark) plan.

There's a lot of middle ground between $7 and $72k. Your quote explains it perfectly though. They flat out can't because the accounting and reporting is badly designed and incapable of providing (near) real-time data.

IMHO the easiest solution to this is government regulation. If you set a budget for a pay-what-you-use online service there should be legislation forbidding companies from charging you more than that.

I also find it (sort of) hilarious they can magically lock the whole thing down once payment fails, but not before the CC is (presumably) maxed out. Lol. Talk about a good deal for Google.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#292

Post by lathiat »

salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

By contrast I hear stories of AWS doing this quite often for one-off mistakes (crediting thousands of dollars). It doesn't make much sense to me not to consider well-intentioned requests for this sort of thing.

Especially if you consider the dollar value of all those approvals and the business you might lose to some other platform and/or hesitance people will have to use those platforms for such things in the future.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#293

Post by [deleted] »

JimDabell wrote: post #24
walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> this post quite honestly just makes me smirk. It seems pretty callous to laugh at somebody else’s $72K misfortune, especially as they took reasonable steps to set a budget on the platform.

[deleted]

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#294

Post by aidenn0 »

oblio wrote: post #238
steren wrote: post #229

(Cloud Run PM here) I am sorry for the experience described in the blog post, we could definitely be better at bill management. I am glad that it worked out in the end and the customer was not required to pay for the bill. Based on this experience, we decided to lower the default value of "max instances" to 100 for future deployments. We believe 100 is a better trade off between allowing customers to scale out and pr…

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

My guess is that the billing logic is separate from the application logic. There's probably a delay between the two and mostly one-way communication.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#295

Post by 1337shadow »

scarygliders wrote: post #61
walrus01 wrote: post #30

Earlier quoted context omitted.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again. It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. It's not callous - it's the p…

I quite agree with everything you've said in this and your other post. My development environment? : My own dual-booting Windows/Linux PC with 32G RAM and a few TB of SSD. Not to mention the Nvidia RTX graphics card for gaming... I either spin up a VM to test stuff, or spin up a Python virtualenv. Postgresql also running on this machine. Whatever's needed. Need to emulate Stuff Happening From Different Servers? Why j…

FaaS is what they call Serverless I guess, anyway, that seems a step backward to me, like going from FastCGI back to CGI, and somehow they market it as "progress".

At least, they could be using OpenFaaS or something, or even free software like firebase such as kuzzle.io or Mozilla Kinto.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#296

Post by pontifier »

sterlind wrote: post #209
pontifier wrote: post #198

Earlier quoted context omitted.

Lol. I love it. I moved to a state I'd never considered because it had the largest, cheapest building in the US. It's 220,000 square feet, but I've lived in a tent out back for the last 6 months because I can't get an occupancy permit, it's not zoned residential, and I refuse to pay rent on an apartment.

You live in a 220,000 square foot building?! Is this an abandoned missile silo or something? I want to know more.

It's the old headquarters of Varco Pruden. They manufactured steel buildings, and there are long, wide manufacturing bays with overhead cranes. You can see much more on my YouTube channel. I've got a few videos of different areas.

This is the first video I made there showing a bit of the size: https://youtu.be/qdkLzioUiAE

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#298

Post by ggthrowaway2020 »

dijit wrote: post #242
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

> but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. Literally did this my first week when trying out GCP for my company. It is entirely possible and documented (with code): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

I'm not sure if you meant it this way, but your tone makes it seem like the parent just needs to "read the docs".

Unfortunately for all of us, your solution doesn't work, per the huge disclaimer on the page that says those alerts can be days late. You can rack up an almost unlimited $ bill in hours.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#299

Post by donmcronald »

Alupis wrote: post #200
sudcha wrote: post #182

Earlier quoted context omitted.

OP here. Bankruptcy fear was real at the time. Google has at least a few thousand lawyers on payroll. They probably also have a process of handling delinquencies and sending them notices. A quick look at the lawyer fee to just manage the case, let alone fight it, is enough for bootstrapped company to raise hands. +1 to bad actors possibility. I shared this with Google team, I'm not sure what they have done since. We…

All true, and good points you raise. However, Google's army of lawyers costs them real money, where your bill is largely made up numbers. Perhaps the true cost is still enough to warrant sic'ing their lawyers on your company. Even in that situation, a wait-and-see approach is still pretty advisable. The worst case scenario was already known to you - bankruptcy. Nothing Google or their lawyers do would change that wor…

> Even in that situation, a wait-and-see approach is still pretty advisable. The worst case scenario was already known to you - bankruptcy.

You could even go scorched earth, represent yourself, and drag it out as long as possible. "Your honor, I'm a free man on the land and all I was doing was travelling the information super highway. I'm not bound by your laws!" Haha.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#300

Post by daveidol »

asciimike wrote: post #191
NegativeLatency wrote: post #140

Earlier quoted context omitted.

Probably hurt revenue ;)

As a former App Engine PM who spent a lot of time with billing/quotas (though, not the one who deprecated this feature), it's likely due to some combination of: - hard limits caused downtime more often than they prevent these blog posts - hard limits were inconsistently enforced, even within GAE - platform wide quota notifications were implemented (reached "GA"), leaving the question of "how a developer wants to hand…

I gotta say that seems like a bad reason to remove the feature. If someone intentionally set a hard spend limit - hit it - and their service went down because of it that's not Google's fault. The simple solution for that customer is to just turn off or increase the limit.
Post reply on HN