Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#311

Post by betwixthewires » Fri, Dec 11, 2020, 1:48 AM UTC

sudcha wrote: post #267

tunesmith wrote: post #224

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

Yeah, that would be a very good idea. The way it is stated gives that impression, and helping people how to understand how to resolve an issue like this would be priceless IMO.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#312

Post by tbodt » Fri, Dec 11, 2020, 1:49 AM UTC

sudcha wrote: post #267

tunesmith wrote: post #224

Deeply dissatisfying to read. Ex-Googler uses connections to get his (understandable!) cloud mistake refunded. Every time I read one of these stories, I get more and more convinced I will just simply never use scalable cloud tech for my side projects. I'm not going to risk my family's retirement savings on the all-too-possible chance that a small deep-implication error will cause runaway charges.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

Yes, please make a post sharing your knowledge on how to convince them. I suspect most people would have no idea how to pull this off.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#313

Post by betwixthewires » Fri, Dec 11, 2020, 1:49 AM UTC

aidenn0 wrote: post #290

dsymonds wrote: post #287

Earlier quoted context omitted.

Did you read Part 2, linked from the bottom? It has plenty of technical details.

I didn't realize that was a link; seriously why does every website have to make links look different.

Lol to me it looked like part 2 wasn't written yet, but I clicked it anyway just to check and the page loaded, so I read it. No real downside in getting a 404.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#314

Post by KorematsuFred » Fri, Dec 11, 2020, 1:50 AM UTC

betwixthewires wrote: post #308

cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

In my opinion, and maybe I'm an absolutist about this, the fact that there aren't these guardrails is opportunistic and predatory. Agile, iterative design and testing will inevitably lead to failures, that's the whole point. Marketing a cloud service to developers who need scalable and changing access to computing during that process should take that into consideration.

I do not think the intentions here are to be opportunistic and predatory but inability to empathize with small developers. A large customer will very likely just pay off few hundred thousand dollars extra expenses. It is only individual developers who are at risk here and cloud operators do not have much interest in them.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#315

Post by KorematsuFred » Fri, Dec 11, 2020, 1:52 AM UTC

cogman10 wrote: post #206

salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

There are already such features but a lot of indie developers are lazy to configure their infra properly. Now, default low limit does not make sense as it will piss off large customers.

I run so many websites on Google Cloud Run that sometimes I feel I might be abusing them, but I have ensured each of my site has max limit of 2 hosts.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#316

Post by betwixthewires » Fri, Dec 11, 2020, 1:53 AM UTC

dathinab wrote: post #282

donmcronald wrote: post #51

Earlier quoted context omitted.

This is my worst nightmare. Lol. I guess now is a great time to give Azure a shoutout for sitting on their hands for 8 years without so much as a response to the community for half a decade [1]. At least AWS allows using a prepaid credit card so they’ll need to call me if things go haywire. I bet if that $72k charge went through it would have been much harder to get out of. “Sorry, we don’t have the money” is a much…

But then consider following hypothetical but possible scenario: Sorry until you pay, no more Amazone services for your company. Now you must move to a new cloud provider (or make a new company). Oh, wait they now interchange (bad) customer information to better find fraud and you just got marked at "owning a lot to amazon" so no cloud for you anymore at any provider. Now you want to buy your own hardware. So you need…

I see no reason why such an arrangement couldn't be optional. Different projects, teams and people have different needs, cloud computing services are marketed specifically on this point. It makes no sense that there isn't even an option in Firebase or AWS to immediately stop services over a certain amount. The current a situation is ripe for lawsuits IMO.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#317

Post by betwixthewires » Fri, Dec 11, 2020, 1:57 AM UTC

foolmeonce wrote: post #268

treeman79 wrote: post #119

Earlier quoted context omitted.

Done correctly they save a lot of IT time. Seem companies hire five 6 figure people to try and cut amazon bill by a couple of grand a month. Never understood spending 50-100k a month to maybe save 5k

>Done correctly they save a lot of IT time Not really, computing done correctly is about avoiding all of the pitfalls and finding ways to get zero cost benefits, free computation out of necessary redundancy, etc. Selling cloud computing is about creating options around every pitfall and finding ways to charge for every mitigation that will be necessary and charge for redundancy in the mitigation strategy for the miti…

They're very useful if you're testing a concept, need agile scaling of computational power, or are just starting a service and don't want to / can't invest the capital in dedicated hardware. I agree with you on your last point though, making your service entirely dependent on these services makes you little more than a franchise and is a potential vulnerability if you ever compete with any important existing service. It probably isn't a good idea for a mature or rapidly maturing business to rely heavily on these services.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#318

Post by infinitelooper » Fri, Dec 11, 2020, 2:04 AM UTC

The algorithm this ex-googles came up with is hilarious, hard to believe this guy came out of google. I thought they were big on algorithms in their interview process? Maybe I should consider putting in an application. If I was interviewing an applicant and they put down the marker after producing that in a whiteboard session they would have quite a bit of work to do in order to get to the next interview phase. Not only did ex-googler not catch those mistakes, his team of 7 actually sat down and coded it without spotting the problem.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#319

Post by betwixthewires » Fri, Dec 11, 2020, 2:08 AM UTC

asciimike wrote: post #199

cesarb wrote: post #150

Earlier quoted context omitted.

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to…

How does this actually work? It clearly can't be forever, since any non-zero dollar amount * infinity months is infinity dollars, which is going to reduce the budget below zero since any non-infinite number minus infinity is less than zero... thus locking it immediately. Even if we say "you get N months of storage before we delete it" and subtract N * current storage cost/month, what happens after you're locked out o…

Off the top of my head I'd say that if you're budgeting for storage, the max you can afford for the time period that you'd need to recover data in the event of a budget overrun taking into considerations the delay time for notifications would be a way to calculate that. And that sounds like something that is reasonable to put on the customer to calculate.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#320

Post by betwixthewires » Fri, Dec 11, 2020, 2:14 AM UTC

jyu wrote: post #272

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

There is an easy explanation: It's hard to build this feature, there is no pressing demand from upper management, it's easier to get promoted doing other simpler projects. Think about what a real time snapshot means: you need to know how much of all the services are being used, project that in the future and compute the costs.

Really, it is a bit disappointing to see a bunch of engineers in this thread talking like this is some monumental, borderline unsolveavle problem. The solution is pretty easy to figure out, even taking into consideration different needs of different customers. The implementation might not be trivial, and legal liability questions might have to be considered beforehand, but the problem is not that hard.