Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

141–150 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#141
post #11

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

You forget your own cost here. A full-time system administrator costs more than 72k a year.

Learning/Administering AWS/GCP/Azure costs time and therefore money too. Maybe less money, maybe more money than doing things yourself, depending on what you're doing. But you shouldn't disregard such costs.

I've seen enough buddies spending enormous amounts of time doing AWS devops on top of paying the AWS premium when they could have gotten away easily with a less than a handful of VPS (+ optionally $100/month worth of cloudflare as a CDN).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#142

To put it into perspective: You give me $72K and I'll set you up a 1PB replicated storage infra with a total of 100+ available CPU cores and half a TB RAM. I saw people burning through cash in the cloud, which makes you wonder weather money is any concern at all.

Because electricity is free. And internet is also free. And the rooms to put the servers are also free. A/C is free. And backup generators are free. And diesel is free.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#143
post #67

Earlier quoted context omitted.

AWS pricing is not obscure, it's just not for you. So in that sense, you are correct to not see a reason to move to the cloud, but your advice does not apply to everyone. And I don't believe they make "more money" that way at all. AWS margins are either very low or very high, and the higher margins and prices tend to be the "simpler" ones: packaged, managed products such as Redshift that are billed on fewer tiers and…

AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.

> There is a massive secondary consulting market because of AWS’s price obscurities.

There is a massive secondary consulting market because the enterprise market is addicted to secondary consulting. This secondary consulting market includes AWS pricing because it includes pretty much any IT service the target market might be interested in.

A rational need for decomplexification isn’t necessary to explain the existence or coverage of enterprise secondary consulting, IT or otherwise.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#144
post #86
post #58

Earlier quoted context omitted.

For AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake…

So you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

> and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

This is HN, many of us are solo founders with no coworkers or employees. Also how could a "friend" pull the plug? If it was a physical server running in your house maybe, otherwise you can't really give them access to your AWS account with all your private clients data in there.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#145
post #106

Earlier quoted context omitted.

Yeah, but it just shows my database cost which is higher than is listed as far as I can tell.

Could it be listed "hourly" and you're charged "daily"? Add in VAT (equal to 25% in some countries) and you match the 30 times higher than expected charge.

https://azure.microsoft.com/en-us/pricing/details/sql-databa...

Basic tier, 5 DTUs, 2 GB is listed as ~$4.8971/month or $0.0068/hour on this page. Extra storage would cost more but is not available for the basic tier.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#146

Earlier quoted context omitted.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

> They want to suck the maximum money from consumers before they realize.

This is a naive understanding of how corporations like Google and Amazon work. Bad will and using gym membership tactics aren't how they scale or make money. Getting you to confidently try things knowing you won't get charged (the reason they have those free tiers) so you'll get your company, your start-up, your next side project on it is much better for business.

It's a miss that things like this aren't implemented and widespread, not by design.

> It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them.

I'm not in this space, but from my observations:

- Each service has a different billing model and metering model. Most likely this data is held by the service. I'm familiar with AWS so I'll use them as an example. I'd wager only DynamoDB or only Lambda (the service owners) know how much of those services you've consumed

- Billing is most likely reconciled asynchronously after collecting all data from all services by an entirely different department with knowledge of payments and accounting

- GCP, AWS, Azure launch 50+ services a year

- Each large customer most likely has a special rate. I bet Samsung or Snap pay an entirely different set of rates than the normal customer. There are thousands of these exceptions

- Cutting your service off when your over the limit is an incredibly complex set of edge conditions. Your long running instance hosting your critical service is shut off because of experimenting on a new ML workflow?

Even with only the above I can see the difficulty in globally limiting your spending limit at an accurate level. I know there are features for both AWS and GCP and they try.

It's easy to stand on the sidelines and handwave away technical complexity at scale, but I'd encourage you to give all of these providers a more charitable view, at least on this topic.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#147
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

AFAIK digitalocean has notification if you go over user defined limit.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#148
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

Probably because it's not so simple on the backend. I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update. Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time. It would be a great feature, but I can imagine it being very complex. It's als…

This is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount.

If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get a bill for $1000.

With an actual hard limit you create a financial incentive for the provider to minimize this overrun. Yes it might be difficult to fix but I assure you, if hard limits existed, the technical issues would be solved soon enough because now there's a reason to invest in a solution.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#149

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

> Most apps don't need scaling anyway and if you do, just put that app on bare metal fitting your requirements.

Most important !

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#150
post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

> Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action.

A better option would be to automatically reduce the budget by the amount it would cost to keep the storage forever. If doing that would reduce the budget to zero, do not allow increasing the amount of storage. That is: assume the storage will not be deleted, and budget according to that.

Post reply on HN