Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#121

Post by lights0123 » Thu, Dec 10, 2020, 6:45 PM UTC

sandGorgon wrote: post #112

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

> There is a delay of up to a few days between incurring costs and receiving budget notifications. Due to usage latency from the time that a resource is used to the time that the activity is billed, you might incur additional costs for usage that hasn't arrived at the time that all services are stopped. Following the steps in this capping example is not a guarantee that you will not spend more than your budget.

This looks like it has the same problems as the post, because it also relies on those budget alerts that can happen a long while after you've exceeded them.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#122

Post by walrus01 » Thu, Dec 10, 2020, 6:45 PM UTC

dang wrote: post #110

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Please don't be a jerk on HN, especially in response to someone else's misfortune, even if they brought it on themselves. Maybe you don't need to treat these people better (though why not?) but you owe the community better if you're posting here. If you wouldn't mind reviewing the site guidelines and taking the intended spirit to heart, we'd be grateful. Note these ones: " Be kind " and " Please don't sneer " https:/…

Thanks for the feedback. I almost certainly shouldn't have included the part about the smirk, and I can definitely see how that could appear to be making fun of somebody else's misfortune. And the rest of it could have been phrased in a more diplomatic way.

For what it's worth it wasn't intended personally at the person who almost incurred the $72k bill, but more at the general concept of test/beta software gone rampant and out of control in an environment where billing has no limits. I think we've all tested some sort of software in development environments that caused havoc - but up until very recently it's been hard for that to immediately begin causing real world financial consequences...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#123

Post by modeless » Thu, Dec 10, 2020, 6:50 PM UTC

sandGorgon wrote: post #112

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

"Following the steps in this capping example is not a guarantee that you will not spend more than your budget."

"Resources [...] might be irretrievably deleted."

Also it's not automatic, you have to manually write code to do it, and test it, and make sure not to break it.

A reasonable implementation of this feature would be built into the console, guarantee a maximum spend, not require writing your own fallible code, and provide an option to preserve storage (at normal cost) so that all your data isn't deleted when your compute/API stuff is shut down.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#124

Post by AngusH » Thu, Dec 10, 2020, 6:52 PM UTC

jasonpeacock wrote: post #102

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

But we've had disk quotas before that mostly worked?

If anything it seems an easier problem than processor time.

I recall disk quotas on shared systems at university back in 1998 and I'm sure they existed before that.

Two thresholds IIRC, one at which you get a warning, second at which you can't write any further and the disk write operation fails.

I don't think they deleted files, it was just you couldn't write more than [quota] bytes to your disk.

Is there something particular about cloud based systems that prevent this from working?

ie. is this a specific problem with distributed storage?

edit:tone

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#125

Post by com2kid » Thu, Dec 10, 2020, 6:53 PM UTC

mlboss wrote: post #94

I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.

I built my startup on a combo of DO and Firebase.

If I knew something was going to take more than a couple dozen milliseconds to run, it was built on the DO droplet.

Why would I pay by the CPU second for something that is taking a lot of CPU seconds? That billing model doesn't make sense.

For my super quick REST endpoints, yeah, all on Firebase, the convenience of writing + deploying makes it an obvious win. (Unless something goes wrong, debugging Firebase functions is not fun...)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#126

Post by reddit_clone » Thu, Dec 10, 2020, 6:56 PM UTC

walrus01 wrote: post #41

onion2k wrote: post #36

Earlier quoted context omitted.

Dell 1U servers on eBay right now for under $500 with 128GB of RAM In part 2 the author says "Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144". In other words, that $500 server is several times more expensive than it would have been if Firebase and GCP had saner defaults.

Yes I would agree that having automatic-scaling set to effectively infinite by default is not the best choice for the end user who is paying. But for the cloud operator, when somebody's runaway application results in a $15,000 bill that has to be paid, sure... As to whether letting people's runaway things scale up infinitely is an intentional choice, I couldn't say.

For me the needle swings towards Malice, away from Mistake on this one. At the very least callousness

Add to the long list of disappointments at humanity:

- Late fees were a big part of BlockBuster's business.

- Police departments factor traffic fines into their budget.

- Thousand other dark patterns that are unethical but not illegal.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#127

Post by Alex3917 » Thu, Dec 10, 2020, 6:57 PM UTC

dang wrote: post #110

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Please don't be a jerk on HN, especially in response to someone else's misfortune, even if they brought it on themselves. Maybe you don't need to treat these people better (though why not?) but you owe the community better if you're posting here. If you wouldn't mind reviewing the site guidelines and taking the intended spirit to heart, we'd be grateful. Note these ones: " Be kind " and " Please don't sneer " https:/…

> Maybe you don't need to treat these people better (though why not?)

IMHO the best argument for 'why not' would be that it's generally unethical to deploy software without first taking the time to read the manual and understand how your dependencies work. In this case the system wasn't live and the costs of this fuckup were solely externalized onto Google, which is fine because it was in large part their fault anyway. But when dealing with production deployments, this same behavior often results in users having all their private information leaked or deleted.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#128

Post by Paul-ish » Thu, Dec 10, 2020, 7:07 PM UTC

greatgib wrote: post #64

serial_dev wrote: post #23

Earlier quoted context omitted.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

The fact that the dashboards and alerts have a delay sounds like there might be difficult consistency stuff going on. Many nodes need to coordinate their usage and billing. It may be a difficult problem, but solving billing problems might not really motivate anyone at the company. It's not a "cool" problem for engineers and not profitable for product.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#129

Post by bjarneh » Thu, Dec 10, 2020, 7:07 PM UTC

donmcronald wrote: post #73

bjarneh wrote: post #54

Earlier quoted context omitted.

I'm sure that was an important factor, but wouldn't this in any case just be a bill to a startup with no money on hand? It's hard to make companies (without money) economically responsible for anything I guess, it even seems hard to make companies with money responsible sometimes.

I can’t set a limit like that on my CC, so the first charge for $5k would have cleared meaning it would have run way longer and racked up way more usage. I’d bet you my computer I would have been out at least the $5k that cleared.

You're probably correct. It really makes you think twice about setting up some of those cloud services without a hard limit cutoff.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#130

Post by [deleted] » Thu, Dec 10, 2020, 7:08 PM UTC

sandGorgon wrote: post #112

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

[deleted]