Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#321

Post by donmcronald » Fri, Dec 11, 2020, 2:14 AM UTC

dathinab wrote: post #284

ggthrowaway2020 wrote: post #194

Earlier quoted context omitted.

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution. In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying. We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, thi…

(By now) GCP documentation says billing alerts can be late by days . Yes days not just hours. Totally crazy. EDIT: Link (see blue info box a a bit below the anchor on which the page is opened): https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

> Recommendation: If you have a hard funds limit, set your maximum budget below your available funds to account for billing delays.

But if 1,000,000% lower doesn't work ($7 vs $70k) then...?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#322

Post by howlgarnish » Fri, Dec 11, 2020, 2:16 AM UTC

donavanm wrote: post #239

cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

This. App Engine used to offer hard spending limits, and they were removed with precisely because so many users set them up to shoot themselves in the foot at precisely the worst possible moment.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#323

Post by betwixthewires » Fri, Dec 11, 2020, 2:17 AM UTC

cambalache wrote: post #187

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

That should be illegal, but hey, at least they support noble causes, so let them be. It sounds cynic but this is their game.

To me this is akin to the personal checking overdraft scams banks were running for many years until those practices were made illegal.

There is no engineering hurdle that is a valid excuse for allowing a customer to go over their stated budget by 86 million percent.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#324

Post by AntiImperialist » Fri, Dec 11, 2020, 2:17 AM UTC

ctvo wrote: post #146

greatgib wrote: post #64

Earlier quoted context omitted.

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

> They want to suck the maximum money from consumers before they realize. This is a naive understanding of how corporations like Google and Amazon work. Bad will and using gym membership tactics aren't how they scale or make money. Getting you to confidently try things knowing you won't get charged (the reason they have those free tiers) so you'll get your company, your start-up, your next side project on it is much…

>Bad will and using gym membership tactics aren't how they scale or make money.

Except they do that with their actions.

>Cutting your service off when your over the limit is an incredibly complex set of edge conditions.

Sure! But if they cared about customers as you claim, they'd let users set hard limits, and when one of these mishaps happened, stop the services when their system eventually knows that the quota has been exceeded... and, make the user only pay the hard limit as the maximum amount. If this continues to happen, warn the user that their account will be terminated... and that's that. But they'll never do that.

Most of their clients pay for these mistakes because they don't have the reach or skills to make this a viral social media article to get people's attention and hence get them to forgive the costs.

I'm sure they know how much they make in revenue because of these mistakes and they deliberately don't do anything about it.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#325

Post by howlgarnish » Fri, Dec 11, 2020, 2:18 AM UTC

vp8989 wrote: post #280

donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active." Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period. The only difference is in that case, it suits AWS to do that w…

If you hit a rate limit, the marginal portion of requests exceeding that limit is dropped: if you plot the requests, the graph gets clipped. Bad, but not catastrophic.

If you hit a billing limit, everything beyond that point is dropped, and the graph of requests plunges to zero. You're effectively hard down in prod.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#326

Post by AntiImperialist » Fri, Dec 11, 2020, 2:22 AM UTC

camjohnson26 wrote: post #260

uoaei wrote: post #162

Earlier quoted context omitted.

> It is baffling why cloud providers don't have that option. ...is it? If a lazy dev leaves their corporate account open and you can bill it for their negligence, protected by the contract you already signed, you earn a lot of money. From a purely business perspective, it is stupid(!) to provide a stopgap for that. Edit: to be clear I am not advocating one way or the other. But it is surprising that people are "baffl…

Google is around a trillion dollar company, your $75,000 is a completely immaterial amount to them. Not to mention it would be a one time payment that would drive away customers and lead to bad PR like this post.

Except Google has 4 million customers. If 1% of their customers made a 75k mistake, they make 3 billion dollars.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#327

Post by betwixthewires » Fri, Dec 11, 2020, 2:25 AM UTC

heavyset_go wrote: post #170

bpodgursky wrote: post #113

Earlier quoted context omitted.

There are some cloud services where it's not quite this simple. S3 -- you can't just delete customer data because they hit a billing limit RDS -- not going to drop databases on the 27th of the month Anything with persistent data is going to have to stay alive and accumulate costs. Admittedly these services aren't where the crazy bills come from, but it does make a simple kill switch a bit more complex.

This is a solved problem for every other service out there. You don't just delete the data, you give the customer a few days, weeks, or a month to pay their bill and if they don't, then you delete their data.

The problem with this though is it opens a vector for exploitation: users could just use the grace period to store data for free for a period of time. This can quickly become a heavy financial burden if enough people do it.

You could factor that into the price, but then you're potentially making the price point even more unattractive to users than it already is, and users that are responsible with their budgets would be subsidizing those that aren't. Not a very workable solution.

I'd say a good solution is giving customers the option to stop accruing more storage capacity, and to have a max deadline accounted for in their budget to store data (basically each customer decides whether or not to pay for a grace period).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#328

Post by segmondy » Fri, Dec 11, 2020, 2:28 AM UTC

If there's ever been a case against using GCP or AWS this is it. You better understand those systems or you will be in for the shock of your life when you get hit with this sort of crazy billing. I got $3,000 AWS credit and was terrified of running experiments in case I made a mistake. It should not be hard to set hard billing limits, these companies know how to set hard CPU limits, why can't they do that with billing?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#329

Post by AntiImperialist » Fri, Dec 11, 2020, 2:28 AM UTC

throwaway201103 wrote: post #168

that_guy_iain wrote: post #50

Earlier quoted context omitted.

> That sort of crap is the reason we host all our stuff on root servers. Having just started my own journey into building products for myself, pretty much the first thing I realised with my tech was I need to get dedicated servers instead of cloud just because it costs 100x less. > Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Exactly, if you really want kubernetes coolne…

> kubernetes ... is super easy to setup Can you point me to the super easy setup guide? Because I've tried a few and never gotten it working.

Canonical has microk8s, which you can install as a snap package. Super simple and works great.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#330

Post by mwcampbell » Fri, Dec 11, 2020, 2:30 AM UTC

kordlessagain wrote: post #247

> The next morning on Saturday, March 28th, I called and emailed over a dozen law firms to book an appointment / have a chat with some attorney. OP is a dick.

More likely they were scared out of their mind and frantically trying to do anything and everything they could to handle the situation.