Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

161–170 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#161

Post by phkahler »

Paul-ish wrote: post #128
greatgib wrote: post #64

Earlier quoted context omitted.

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

The fact that the dashboards and alerts have a delay sounds like there might be difficult consistency stuff going on. Many nodes need to coordinate their usage and billing. It may be a difficult problem, but solving billing problems might not really motivate anyone at the company. It's not a "cool" problem for engineers and not profitable for product.

>> The fact that the dashboards and alerts have a delay sounds like there might be difficult consistency stuff going on.

I think that's true. It's easier to measure usage and aggregate that data after the fact than to meter it in real time and stop at a limit. Those are very different things. What happens if you hit the cap while running multiple processes spread across a cloud?

One improvement might be to throttle things as the cap approaches but that doesnt really change the problem at all. Do that and have provider eat any overages should solve it from the user point of view.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#162

Post by uoaei »

serial_dev wrote: post #23
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

> It is baffling why cloud providers don't have that option.

...is it? If a lazy dev leaves their corporate account open and you can bill it for their negligence, protected by the contract you already signed, you earn a lot of money. From a purely business perspective, it is stupid(!) to provide a stopgap for that.

Edit: to be clear I am not advocating one way or the other. But it is surprising that people are "baffled" by this obvious profit optimization.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#163

Post by raphaelj »

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's ever worse for services like AWS Cloudfront.

One of your competitors could just rent a cheap server on OVH with uncapped transfer and incur you $10k in cost in a few hours.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#164

Post by scrollaway »

Closi wrote: post #118
klohto wrote: post #75

Earlier quoted context omitted.

> There is a massive secondary consulting market because of AWS's price obscurities. While that's true, there is consulting market for most things that are complicated. Doesn't mean they are shady. It's simply not for you. You are welcome either to dive in or get a consultant. I promise you though, that AWS pricing isn't difficult once you understand few concepts and know your way around the Cost Explorer. With prope…

> While that's true, there is consulting market for most things that are complicated. Doesn't mean they are shady. It does mean it's not simple though.

Obscure and complex are different concepts. I'm part of that "secondary consulting market" FWIW, so I'd like to think I know a thing or two about it.

Does AWS have high-margin prices? In aggregate, somewhat, but this is mostly driven by the big ticket managed enterprise items: Aurora, Redshift, Quicksight, probably Fargate, etc. A lot of their more popular stuff (S3, Lambda, …) offer incredible value for very little money. EC2 is the exception I believe, because I understand it to be high margin for how popular it is. But EC2 pricing is one of their simplest ones.

Could AWS simplify some of their pricing? Yes, probably. There's always room for optimization. Personally for example I'd like to see their pricing be global rather than different by region (with understandable exceptions for govcloud and china).

Is AWS making its pricing complicated for nefarious purposes? No, there is no evidence to support that.

AWS pricing absolutely is not simple. It's a part of the AWS stack. You need to study AWS's events/signals system to be able to write apps that make the best use of AWS's interconnected stack. You need to study their APIs / SDKs to really understand what you're able to implement. And you need to study their billing systems to understand how to implement apps that run cheaply, and be able to predict potential runaway costs.

It has to be a part of the design. That's why you may want to hire consultants for it: People who understand it better than you do, and will be able to assist you in reducing your costs.

It's just another kind of optimization. Maybe some software engineers don't like it because it hits them where it hurts (the wallet) when they don't do it right, rather than be able to brush it off as they usually do.

It's much easier to ignore the waste produced by, say for example, the 3000 javascript dependencies shipped with the fat, unoptimized electron app they ship on their users' desktops, that do a ton of unnecessary expensive computing; when all that crap is client-side and it's the downstream user's electricity bills and CPU time that's being used.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#165

Post by Alupis »

As J. Paul Getty once mused[1]:

> If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem.

Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature.

Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat the "experiment", but seriously... there's no mechanism at Google to stop a new client from running up a near-$100k bill in a single day?

That's absurd, and should be a learning lesson for Google more than this startup. Some malicious actor could apparently consume hundreds of thousands of dollars of Google resources and "get away" with it.

Wait and see what happens, then deal with it - would be sane advice.

[1] https://www.brainyquote.com/quotes/j_paul_getty_129274

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#166

Post by mrtksn »

sandGorgon wrote: post #112
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

OP claims that the budgets are not real time, they are eventually accurate but if it happens that you spend too fast you may end up with a larger than your budget sum before anything triggers.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#167

Post by altdatathrow »

Do not use hosted cloud services where the implementation creates publicly accessible API keys and each HTTP request results in a charge to your account. A few specific examples are Firebase, Algolia, and AWS Lambda.

All it takes is one programming mistake or one bad actor and you can find yourself in an equally precarious situation.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#168

Post by throwaway201103 »

that_guy_iain wrote: post #50
LordHeini wrote: post #42

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

> That sort of crap is the reason we host all our stuff on root servers. Having just started my own journey into building products for myself, pretty much the first thing I realised with my tech was I need to get dedicated servers instead of cloud just because it costs 100x less. > Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Exactly, if you really want kubernetes coolne…

> kubernetes ... is super easy to setup

Can you point me to the super easy setup guide? Because I've tried a few and never gotten it working.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#169

Post by thanksforthe42 »

csharptwdec19 wrote: post #137
thanksforthe42 wrote: post #116

Earlier quoted context omitted.

Calling programmers "Engineers" is a misnomer. I wish programmers had the prestige it deserved for combining Science, tradition, authority, and art. Engineers are not allowed to use tradition, authority or art. They are restricted to being modern day calculators. Nothing is wrong with either.

The shift from 'Developer/Programmer' to engineer has indeed been part of a push away from creativity towards cookie-cutter work. An interesting analogue would be the Automotive industry; As time progressed, Companies focused more and more on 'engineering' versus art/tradition/etc. But as the industry evolved, "Flashy" vehicles that took risks became moreso either a halo product for a brand, or relegated to Luxury/Bo…

I think that missed the point.

Engineers are applied scientists.

Programmers are not applied scientists.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#170

Post by heavyset_go »

bpodgursky wrote: post #113
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

There are some cloud services where it's not quite this simple. S3 -- you can't just delete customer data because they hit a billing limit RDS -- not going to drop databases on the 27th of the month Anything with persistent data is going to have to stay alive and accumulate costs. Admittedly these services aren't where the crazy bills come from, but it does make a simple kill switch a bit more complex.

This is a solved problem for every other service out there. You don't just delete the data, you give the customer a few days, weeks, or a month to pay their bill and if they don't, then you delete their data.
Post reply on HN