Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

21–30 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#21

Post by wrkronmiller »

walrus01 wrote: post #20
skrebbel wrote: post #17

Earlier quoted context omitted.

> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.

fine, buy two identical ones. and set up proper backups. or even make the backup a hot-spare. if it's a test/development system that's meant to possibly break, you shouldn't be driving anywhere at 0300 in the morning anyways.

That implies you are going to be running prod in the cloud. Unless you're developing against purely synthetic data, the data transfer costs are potentially astronomical.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#22

Post by jsiepkes »

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#23

Post by serial_dev »

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It is baffling why cloud providers don't have that option.

I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users.

By the way, the same goes for some companies, too, just the threshold would be different.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#24

Post by JimDabell »

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> this post quite honestly just makes me smirk.

It seems pretty callous to laugh at somebody else’s $72K misfortune, especially as they took reasonable steps to set a budget on the platform.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#25

Post by daneel_w »

skrebbel wrote: post #17
walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.

It's anecdotal, but I'm convinced I'm not alone here: we've had more Amazon-related failures/outages in 3 years with AWS than we had in 4 years of colo before heading to the cloud because of the exact fear you described.

Even a cloud setup needs good management and contingency planning, and in absence of such it can fail just as hard as a colo setup.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#27

Post by gruturo »

LeonM wrote: post #11
throwaway7281 wrote: post #8

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

And for half the use cases you still need one. Unless you go full SaaS (which may or may not be an option depending what you are doing, and what's on offer in that field), you still get stuff which needs to be administered, updated, patched, etc. Maybe not the OS layer (or maybe even that), maybe not the DB (but then it might cost more), but you're not getting away from that.

You only really start saving at some scale (get a small core of cloud-literate admins, and now you can have them run thousands of systems for effectively no incremental cost)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#28

Post by thrower123 »

I'm not sure I want to know how much Azure and AWS revenue comes from people spinning up test VMs or a kubernetes cluster to work through a training, and then forgetting to turn it off.

I've spent thousands extra this year because people stood up 4 MB SQL databases and let them default to charging by vCores instead of DTUs.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#29

Post by jcelerier »

LeonM wrote: post #11
throwaway7281 wrote: post #8

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#30

Post by walrus01 »

JimDabell wrote: post #24
walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> this post quite honestly just makes me smirk. It seems pretty callous to laugh at somebody else’s $72K misfortune, especially as they took reasonable steps to set a budget on the platform.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again.

It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring.

It's not callous - it's the perspective of the people who are responsible for the hypervisors that run underneath the VMs and services that cause some of these massive billing outrages.

Post reply on HN