Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

31–40 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#31

Post by spacemanmatt »

pettycashstash2 wrote: post #2

Thank you for sharing. I was actually thinking of using fire base for my project. They make us so easy to sign up for free tier. Awaiting to see what happens in part 2

I recently put the (soft) kibosh on a project in my stable trying to switch to FireBase at the last minute.

It looks attractive but the business aspects are frankly frightening, and I'm not even talking about the risk of a large bill. Getting your metrics 24h late sounds like a deal killer for me. So much for observability!

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#32

Post by akh »

> Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144. Had we chosen concurrency of “1” request, we probably wouldn’t have even noticed the bill.

> If you count the number of pages in GCP documentation, it’s probably more than pages in few novels. Understanding Pricing, Usage, is not only time consuming, but requires a deep understanding of how Cloud services work. No wonder there are full time jobs for just this purpose!

Great write-up - thanks for sharing @bharatsb! As you say, cloud pricing has become too complex for developers to understand quickly (they want to ship features, not calculate costs). Infra-as-code is great, but it has made it even harder to understand which code/config option costs what. `terraform apply` is like a checkout screen without prices.

We're trying to solve this problem with infracost.io, initially looking at Terraform. It would be interesting to get your feedback on whether such an approach might have helped you? Probably not as it doesn't look like you were using Terraform?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#33

Post by walrus01 »

daneel_w wrote: post #25
skrebbel wrote: post #17

Earlier quoted context omitted.

> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.

It's anecdotal, but I'm convinced I'm not alone here: we've had more Amazon-related failures/outages in 3 years with AWS than we had in 4 years of colo before heading to the cloud because of the exact fear you described. Even a cloud setup needs good management and contingency planning, and in absence of such it can fail just as hard as a colo setup.

https://www.bbc.com/news/technology-55087054

https://www.seattletimes.com/business/amazon/amazon-web-serv...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#35

Post by z3t4 »

skrebbel wrote: post #17
walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.

Just because you use "the cloud" doesn't mean you don't need backups. "the cloud" also have downtime and other failure. When deploying to the cloud you have to factor in the cost for moving to another provider if/when it will be needed.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#36

Post by onion2k »

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Dell 1U servers on eBay right now for under $500 with 128GB of RAM

In part 2 the author says "Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144". In other words, that $500 server is several times more expensive than it would have been if Firebase and GCP had saner defaults.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#37

Post by LeonM »

nlitened wrote: post #19
LeonM wrote: post #11

Earlier quoted context omitted.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Yeah, but the absence of system administrator has just cost these guys 72k for several hours.

I'm just trying to explain that server cost is more than just the hardware.

In most cases cloud computing is actually still a very cost effective solution to infrastructure. But with infinite scalability also comes responsibility.

In the case of the OP, had they had their own hardware they would have noticed that had written bad code (it would have crashed or become very slow at least), but the cloud just scaled up and processed their code.

I'm not trying to defend Google in this case. Billing 72k when a 100 USD limit is set sounds like a scam.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#38

Post by k__ »

jsiepkes wrote: post #22
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.

What about throttling?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#39

Post by LeonM »

jcelerier wrote: post #29
LeonM wrote: post #11

Earlier quoted context omitted.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year

You confuse cost with salary.

Someone who makes 35k a year costs much more. Think about office space, training, insurance, payroll costs etc.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#40

Post by FastQ »

I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for sharing.
Post reply on HN