Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#41

Post by walrus01 » Thu, Dec 10, 2020, 12:20 PM UTC

onion2k wrote: post #36

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Dell 1U servers on eBay right now for under $500 with 128GB of RAM In part 2 the author says "Had we chosen max-instances to be “2”, our costs would’ve been 500 times less. $72,000 bill would’ve been: $144". In other words, that $500 server is several times more expensive than it would have been if Firebase and GCP had saner defaults.

Yes I would agree that having automatic-scaling set to effectively infinite by default is not the best choice for the end user who is paying.

But for the cloud operator, when somebody's runaway application results in a $15,000 bill that has to be paid, sure...

As to whether letting people's runaway things scale up infinitely is an intentional choice, I couldn't say.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#42

Post by LordHeini » Thu, Dec 10, 2020, 12:21 PM UTC

That sort of crap is the reason we host all our stuff on root servers.

Even trying to read the amazon pricing for their instances, hours and what not, drives me insane.

Seems this is done on purpose. no wonder they make so much money with it.

So i have never seen a reason to move any stuff to the cloud.

Just grab a dedicated server for a few bucks and put a bunch of docker containers on those.

Its way cheaper, usually not more complicated. Just use a CI with Gitlab runners or whatever and be done with it.

Most apps don't need scaling anyway and if you do, just put that app on bare metal fitting your requirements.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#43

Post by raverbashing » Thu, Dec 10, 2020, 12:21 PM UTC

LeonM wrote: post #11

throwaway7281 wrote: post #8

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Which is moot if you're just going to burn the 72k by shooting yourself in the foot.

And the way these cloud services go, the 72k was only detected because it was an one-off event. Turn that into a base-level inefficiency that costs that over a year and what have you then.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#44

Post by senko » Thu, Dec 10, 2020, 12:22 PM UTC

skrebbel wrote: post #17

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

> The costs would be absolutely fixed and known. Until the server breaks and you have to drive over in the middle of the night and try to replace it but the only available server right now is a shitty one and oh shit only half the backups work cause the onsite backups are fried too etc etc etc. There's many good arguments against high-level BaaS such as Firebase but I'm not sure that "colo is cheaper" is one.

> I'm not sure that "colo is cheaper" is one.

It absolutely is (cheaper, and a good argument). As an example: we're in the process of switching from Digital Ocean to Hetzner for a project, that will increase infrastructure performance (roughly memory/cpu/storage) by 4x and decrease costs by 4x. And no driving to the colo center is neccessary, as it's their dedicated server, so their on-site engineers do the hardware replacement.

Also, if you are not okay with your site being down for a few hours, you can always buy two, like you would with a sensible cloud setup. It'd still come up way cheaper (+ get more perf if you can do load balancing for your usage).

Also, I don't look at it from "colo is cheaper" point of view. To me, it's "I can have several times more performance and hire a full time sysadmin to worry about it, for the same price".

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#45

Post by herendin2 » Thu, Dec 10, 2020, 12:24 PM UTC

k__ wrote: post #38

jsiepkes wrote: post #22

Earlier quoted context omitted.

Not only development but also running in production. You can configure alerts but you can't configure a hard limit. Thats just insane. That makes working with GCP like playing with fire.

What about throttling?

Nice to have, but people want a throttle that shuts off dead at a certain number of dollars

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#46

Post by chefkoch » Thu, Dec 10, 2020, 12:24 PM UTC

jcelerier wrote: post #29

LeonM wrote: post #11

Earlier quoted context omitted.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Aha what ? Where I live (Bordeaux, France) a quick glance through the job offers for full-time sysadmin are between 25 and 35k€ / year

France seems to pay IT quite bad. In Germany nobody would work for this in metro area.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#47

Post by gonzo41 » Thu, Dec 10, 2020, 12:26 PM UTC

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars.

If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a raspberry pi on your desk.

But really the first lesson you should learn in any cloud setup is Billing Alarms :)

If you're doing ML or CV work then it's probably cheaper to build on the desktop and port to cloud once you understand what the workloads are.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#48

Post by [deleted] » Thu, Dec 10, 2020, 12:28 PM UTC

FastQ wrote: post #40

I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for shar…

[deleted]

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#49

Post by ZephyrBlu » Thu, Dec 10, 2020, 12:29 PM UTC

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Probably because it's not so simple on the backend.

I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update.

Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time.

It would be a great feature, but I can imagine it being very complex. It's also probably cheaper for them to just wave away excess usage like this instead of building out a solution.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#50

Post by that_guy_iain » Thu, Dec 10, 2020, 12:30 PM UTC

LordHeini wrote: post #42

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

> That sort of crap is the reason we host all our stuff on root servers.

Having just started my own journey into building products for myself, pretty much the first thing I realised with my tech was I need to get dedicated servers instead of cloud just because it costs 100x less.

> Just grab a dedicated server for a few bucks and put a bunch of docker containers on those.

Exactly, if you really want kubernetes coolness to act cloud like, install kubernetes it's free and is super easy to setup.

And with the cost savings you can literally buy multiple spare servers and with kubernetes using them all while keeping the usage low allowing to scale up new nodes if needed.