Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#191

Post by asciimike » Thu, Dec 10, 2020, 9:15 PM UTC

NegativeLatency wrote: post #140

asciimike wrote: post #133

Earlier quoted context omitted.

Extremely technically, the only GCP product that had this feature was App Engine Standard v1, but looks like it's deprecated as of the end of 2019 ( https://cloud.google.com/appengine/docs/managing-costs#chang... )

Probably hurt revenue ;)

As a former App Engine PM who spent a lot of time with billing/quotas (though, not the one who deprecated this feature), it's likely due to some combination of:

- hard limits caused downtime more often than they prevent these blog posts

- hard limits were inconsistently enforced, even within GAE

- platform wide quota notifications were implemented (reached "GA"), leaving the question of "how a developer wants to handle this" to the developer, not the platform

- maintenance burden

The "I bankrupted my startup by running tests in an infinite loop" blog posts happen ~once a year, while the number of customers (including internal teams!) who inadvertently went down because of this quota was staggering. I feel like I used to see one a week, at least. Most often someone on the team was like "oh I'm going to turn this down to zero because we don't want to spend any money during development", never told anyone, and then they go live and they forgot to turn the knob back up (or didn't properly estimate traffic/costs and set it too low).

I can tell you it hurts revenue a lot more when a large customer goes down for 15 minutes due to quota issues and their usage drops to zero (both in terms of revenue and customer credibility) vs when tiny developer accidentally blows through 10k in a month and we refund it (since, obviously, the providers cost is a lot less than that).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#192

Post by throwaway7281 » Thu, Dec 10, 2020, 9:17 PM UTC

iooi wrote: post #142

throwaway7281 wrote: post #8

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

Because electricity is free. And internet is also free. And the rooms to put the servers are also free. A/C is free. And backup generators are free. And diesel is free.

I was stretching the point, but what amazes me is the amount of stuff people want to do vs. the amount of equipment they throw at the problem.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#193

Post by jasonpeacock » Thu, Dec 10, 2020, 9:17 PM UTC

modeless wrote: post #156

jasonpeacock wrote: post #102

Earlier quoted context omitted.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost. Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.

But storage costs continue to add up even when you're not accessing them - there's a cost to storage existing which continues to accrue with time.

When there's no budget left, what do you do with those accruing costs for existing storage?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#194

Post by ggthrowaway2020 » Thu, Dec 10, 2020, 9:19 PM UTC

sandGorgon wrote: post #112

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution.

In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying.

We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, this was root-caused to us, because we were submitting jobs continuously). They pointed to their TOS and said they don't guarantee on-time delivery of the alert.

I had to write up a blog post with fancy graphs and prepare it for social media before they finally agreed to eat the bill.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#195

Post by salmonlogs » Thu, Dec 10, 2020, 9:23 PM UTC

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life.

Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of tickets and attended many, many meetings to make this happen.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#196

Post by tony » Thu, Dec 10, 2020, 9:24 PM UTC

Google eventually forgave the bill in Part 2: https://blog.tomilkieway.com/72k-2/

> Google let go of our bill as a one time gesture!

Thank goodness.

And it looks like it had to do with not understanding the API / system on the first order, IMO.

This hit me hard a few months ago with CloudFront invalidations on AWS. I check billing and the things at 30usd in a single day, from a norm of Aside: Amazon has some nice features for rule-based alarms on accounts so when you spend more than X dollars, you get an email.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#197

Post by wikibob » Thu, Dec 10, 2020, 9:25 PM UTC

benlivengood wrote: post #155

a-priori wrote: post #148

Earlier quoted context omitted.

This is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount. If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get…

It's also a mostly solved problem because advertisers have budgets and it's common to implement globally distributed budget servers to avoid showing more ads than the advertiser paid for, despite tens of thousands of individual web servers needing to know which ads in their inventory have budget left. It's a fun exercise similar to global rate-limiting/load-balancing.

That is fascinating.

If you have the time could you (anyone feel free) talk a bit about how you would implement a globally distributed budget?

I can imagine a few simple options, but they all seem to have significant shortcomings.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#198

Post by pontifier » Thu, Dec 10, 2020, 9:26 PM UTC

kabirgoel wrote: post #174

Alupis wrote: post #165

As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat t…

One of my favorite quotes of all time. J. Paul Getty was quite the weirdo. His Wikipedia article is worth a look, especially the section on his frugality.

Lol. I love it. I moved to a state I'd never considered because it had the largest, cheapest building in the US.

It's 220,000 square feet, but I've lived in a tent out back for the last 6 months because I can't get an occupancy permit, it's not zoned residential, and I refuse to pay rent on an apartment.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#199

Post by asciimike » Thu, Dec 10, 2020, 9:30 PM UTC

cesarb wrote: post #150

jasonpeacock wrote: post #102

Earlier quoted context omitted.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to…

How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately.

Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out of all actions because you added an extra GB? Storage APIs cost money to use, so you would get locked out of those too (note that if you're not, people would set arbitrarily low limits and get storage access for free) and couldn't retrieve anything. The only remaining actions are delete (which is free) or raise the quota and do the whole rodeo over again.

Abuse is impossible to ignore at public cloud scale, so "free storage forever" (or even, storage at a one time fixed price) as the fallback isn't a viable option.

Lastly, from an optics perspective, which blog post would you rather see on the front page of HN: "I did something dumb and spent too much money on Cloud" or "Google is holding our data hostage" (or "Google deleted all my data")?

Source: I launched Firebase Storage, which has a GCS bucket that has a hard limit.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#200

Post by Alupis » Thu, Dec 10, 2020, 9:31 PM UTC

sudcha wrote: post #182

Alupis wrote: post #165

As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat t…

OP here. Bankruptcy fear was real at the time. Google has at least a few thousand lawyers on payroll. They probably also have a process of handling delinquencies and sending them notices. A quick look at the lawyer fee to just manage the case, let alone fight it, is enough for bootstrapped company to raise hands. +1 to bad actors possibility. I shared this with Google team, I'm not sure what they have done since. We…

All true, and good points you raise.

However, Google's army of lawyers costs them real money, where your bill is largely made up numbers.

Perhaps the true cost is still enough to warrant sic'ing their lawyers on your company.

Even in that situation, a wait-and-see approach is still pretty advisable. The worst case scenario was already known to you - bankruptcy.

Nothing Google or their lawyers do would change that worst-case outcome, and if Google was aware you literally don't have $72k, and might just declare bankruptcy and walk away, they'll be much more eager to negotiate a more reasonable bill and settle your account. It's exactly as J. Paul Getty said...

Very glad it's being worked out and you will not have to go down that path.