Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

61–70 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#61

Earlier quoted context omitted.

> this post quite honestly just makes me smirk. It seems pretty callous to laugh at somebody else’s $72K misfortune, especially as they took reasonable steps to set a budget on the platform.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again. It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. It's not callous - it's the p…

I quite agree with everything you've said in this and your other post.

My development environment? : My own dual-booting Windows/Linux PC with 32G RAM and a few TB of SSD. Not to mention the Nvidia RTX graphics card for gaming...

I either spin up a VM to test stuff, or spin up a Python virtualenv. Postgresql also running on this machine. Whatever's needed. Need to emulate Stuff Happening From Different Servers? Why just spin up another few VM's - assign them the minimum resources required to get them doing what they need to do, set up your VM network etc. Any decently specced desktop machine can do that, never mind a noisy rack system - considering they're way better and vastly more powerful then the PC's we had 2-5 years before that, which themselves were vastly more powerful than the ones before them, and so on...

Result? Can develop at home to my heart's content, then when it comes to deployment spin up a remote VM on e.g. DigitalOcean and take it from there.

At the end of the day, "sErVeRlEsS" (I just don't like that term, for some reason it rubs me up the wrong way, perhaps because of...) just means "running stuff on someone else's kit" - the same as "tHe ClOuD", so if I'm going to be developing some system & software, I'd rather be doing it locally, setting up whatever's needed to get it running, and once satisfied, deploying it.

Like you, I see either the same people, or new people, simply Not Learning From The Past. There are many good reasons why things were done like they were - developing on a system you own, for example, rather than spinning up all sorts of Cloudy Things or "serverlessy things" right from the start.

Hardware is cheap - you don't need a supercomputer to run the beginnings of your latest Supah Scalable System[tm], you just develop and run it on a reasonably up to date box, and, sure, when you get to the stage where you need more space/bandwidth/whatever, that's the point where you deploy to some Cloudy Thing or SeRvErLeSs Thing.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#62
post #47
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars. If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a r…

Just to expand on this. You can have a hard limit. For AWS, create a role/user that's essentially ~root like access. Make a lambda function that's triggered by a billing alert at your threshold to just turn off things from most expensive to least. So turn of the DB servers. So the apps error out and the users go away.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#63
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Price transparency is the antithesis to the "cloud" and it's current financial success.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#64
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them.

They want to suck the maximum money from consumers before they realize.

For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#65
post #58
post #47

Earlier quoted context omitted.

I wouldn't be too scared. For AWS you get about $0.20 per 1 million requests on Lambda. You can do quite a lot with a single Lambda function. And a million of anything is a lot for a dev. Put a HTTP API Gateway infront of that with a CDN and you're hitting ~ a few dollars. If you don't buy one coffee, or put a 20 dollar note in a book one month. Then you're fine. And if you have to use EC2, just use a t2.micro or a r…

For AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake…

You can trigger events from alarms. And Lambda's only last 15 minutes. So still cheaper than 75K :D.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#66

Earlier quoted context omitted.

From my point of view after doing this for 20 years, it's like seeing the past 12 years of the "put everything in the cloud" era, of new different people repeating exactly the same mistake over and over again. It's like if you lived near a public park with particularly aggressive geese that return every year, and watched new ignorant groups of people get chased by the geese every spring. It's not callous - it's the p…

I quite agree with everything you've said in this and your other post. My development environment? : My own dual-booting Windows/Linux PC with 32G RAM and a few TB of SSD. Not to mention the Nvidia RTX graphics card for gaming... I either spin up a VM to test stuff, or spin up a Python virtualenv. Postgresql also running on this machine. Whatever's needed. Need to emulate Stuff Happening From Different Servers? Why j…

My personal home office development environment at the moment, done on an ultra low budget, is a dell precision t5600 mid tower workstation PC (dual xeon, e5-2630) that I got for $350 with 64GB of RAM in it, upgraded it to 128GB, and put a $150 Samsung SATA3 SSD in. It's small and relatively quiet and sits under my desk tucked in a back corner with just a power cable and a few ethernet cables plugged into it.

Maybe some time in the near future I'll add a 2TB HDD that I have sitting around into it so that I can create VMs that have a 'fast' boot/root disk, and also give them some lvm partitions on a big slow disk.

It's running debian stable amd64 and is set up as a xen dom0 hypervisor, with 768MB of RAM assigned to the dom0 and the rest available for VMs.

The amount of capacity that's available there to create random PV or HVM VMs with as much RAM as I could want, is more than sufficient for my personal needs. If I need anything bigger I'll make it a more formal process and put it on a machine at work.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#67

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and…

AWS's margin is currently 30+% which is massive.

>AWS pricing is not obscure

There is a massive secondary consulting market because of AWS's price obscurities.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#68
post #40

I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for shar…

Go to billing > cost analysis > filter by resource break down. Azure billing analysis is pretty amazing.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#69
At the end of page 2, there is a good ass licking bullshit sentence:

>

He said that as an ex googler and as the beneficiary of a gesture, but this contradict the full history he told us. If the doc and tools were so great, why he felt into this situation?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#70
post #67

Earlier quoted context omitted.

AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and…

AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.

The margin is absolutely not the same across all products.

> There is a massive secondary consulting market because of AWS's price obscurities.

Its. Not. For. You.

AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill.

That consulting market is an optimization market. It's economics at its best.

If you are too small to have to take these things into account regardless, AWS is not for you. You're welcome to use it, but don't be surprised if you end up having to deal with these kinds of things which simply don't exist in the world of flat-price underprovisioned droplets.

Post reply on HN