Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

351–360 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#351

Post by archi42 »

And another day that I'm again totally surprised how some people do "engineering". Basic problems like recursion don't occur to them, they use a technology they don't understand, they are not careful with the number of instances and then burn ~16000 (or close to 2y) of cpu time.

Why not test the algorithm locally, realize the problems and fix them? Why test on 1000 instances, not 1 or 2?

I get the "move fast, fail fast" attitude and why it is deemed beneficial by some, but this was essentially "goto fail".

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#352

Post by teekert »

coddle-hark wrote: post #135
MaxBarraclough wrote: post #10

Earlier quoted context omitted.

> Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.

I can sympathise with some of these stories, like the ones where an overnight DDOS attack racks up a huge unexpected bill, but this one in particular is just a story of gross incompetence and negligence. The guy hacked together some code in a few days and deployed it to a service with unlimited billing without any kind of sanity checks and without even understanding what he was paying for. He’s an ex-Googler, it’s no…

But "The Guy" was under the impression that he could do so because he set sane limits in both GCP and his Credit Card.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#353

Post by teekert »

MaxBarraclough wrote: post #10
bharatsb wrote: post #4

Part 2: https://blog.tomilkieway.com/72k-2/

> Google let go of our bill as a one time gesture! We've seen this happen with similar stories on AWS. Neither platform supports prepayment with a hard limit on costs, and this seems unlikely to change.

Yeah, it seems to be by design, hard to imagine otherwise.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#354

Post by cesarb »

asciimike wrote: post #199
cesarb wrote: post #150

Earlier quoted context omitted.

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to…

How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately. Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out o…

For it to work, obviously the budget has to be per month (for instance, $100/month), instead of an absolute limit. Most of the time, that's what you'd want: if you calculated that what you use will cost $50 each month, setting a budget of $100 per month would give some room for growth while preventing billing disasters (and you can always increase it a bit if necessary).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#355

Post by rightbyte »

betwixthewires wrote: post #331
KorematsuFred wrote: post #314

Earlier quoted context omitted.

I do not think the intentions here are to be opportunistic and predatory but inability to empathize with small developers. A large customer will very likely just pay off few hundred thousand dollars extra expenses. It is only individual developers who are at risk here and cloud operators do not have much interest in them.

I don't know about that. Large customers can almost always find the capital to run their own infrastructure and save on cost. That's not to say that there aren't big customers of these types of services or that certain business models make using them more attractive than maintaining infrastructure yourself, but I would guess that revenue from these sorts of services are largely built on appealing to smaller customers…

When I was a young and naive student I thought I could not charge my debit card under 0$. Got down to -3$ and had to pay a 40$ something fee when I already was out of money.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#356

Post by m1gu3l »

rightbyte wrote: post #355
betwixthewires wrote: post #331

Earlier quoted context omitted.

I don't know about that. Large customers can almost always find the capital to run their own infrastructure and save on cost. That's not to say that there aren't big customers of these types of services or that certain business models make using them more attractive than maintaining infrastructure yourself, but I would guess that revenue from these sorts of services are largely built on appealing to smaller customers…

When I was a young and naive student I thought I could not charge my debit card under 0$. Got down to -3$ and had to pay a 40$ something fee when I already was out of money.

Definitely drank a few 45 dollar lattes in my day. Sucks.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#357

Post by rightbyte »

xwdv wrote: post #301
salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

If I were in this situation I would probably offer 10% of the bill to the employee as a reward for their help.

Sounds like a bribe.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#358

Post by remus »

cogman10 wrote: post #266
donavanm wrote: post #239

Earlier quoted context omitted.

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

> And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. What? Why wouldn't this just be an opt in thing? It could even be tied to the account being used. It's not like AWS accounts are expensive or hard to setup. If a user opts in to the "kill if bill goes too high" and they kill a critical portion of their business, then t…

> What? Why wouldn't this just be an opt in thing?

"Oh cool, I'll set a $1k cap, never gonna spend that on this little side proj." Fast forward a year, the side proj has turned in to a critical piece of the business but the person who set it up has left and no one remembered to turn of the spending cap. Busy christmas shopping period comes along, AWS shuts down the whole account because they go over the spending cap, 6hr outage during peak hours, $20k sales down the pan.

Of course it is technically the customers fault but it's a shit experience. Accidentally spending $72k is also technically the customers fault and also a shit experience. I don't think there is an easy solution to this problem.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#359

Post by freemint »

remus wrote: post #358
cogman10 wrote: post #266

Earlier quoted context omitted.

> And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. What? Why wouldn't this just be an opt in thing? It could even be tied to the account being used. It's not like AWS accounts are expensive or hard to setup. If a user opts in to the "kill if bill goes too high" and they kill a critical portion of their business, then t…

> What? Why wouldn't this just be an opt in thing? "Oh cool, I'll set a $1k cap, never gonna spend that on this little side proj." Fast forward a year, the side proj has turned in to a critical piece of the business but the person who set it up has left and no one remembered to turn of the spending cap. Busy christmas shopping period comes along, AWS shuts down the whole account because they go over the spending cap,…

There exist totally a solution. It is also user hostile enough so it might get's adopted. $cloud_vendor just has to (and probably will) constantly nudge people to loosen the limit. Have a red banner that says, "you spend 3% of your monthly budget already think about increasing it". Also routinely send out emails to remind people. " Black Friday comes up think about increasing your quota " when your service has nothing to do with E-Commerce.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#360

Post by freemint »

howlgarnish wrote: post #325
vp8989 wrote: post #280

Earlier quoted context omitted.

"The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active." Lol what ... this is exactly what happens any time you hit a rate limit on any AWS service. The customers application is "catastrophically interrupted" during its most popular/active period. The only difference is in that case, it suits AWS to do that w…

If you hit a rate limit, the marginal portion of requests exceeding that limit is dropped: if you plot the requests, the graph gets clipped. Bad, but not catastrophic. If you hit a billing limit, everything beyond that point is dropped, and the graph of requests plunges to zero . You're effectively hard down in prod.

Just as it is with bank accounts. Once you run out of money you hit a hard floor.
Post reply on HN