Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#151

Post by renewiltord » Thu, Dec 10, 2020, 7:52 PM UTC

thanksforthe42 wrote: post #114

mikestew wrote: post #100

Earlier quoted context omitted.

I have very little money... Then the harsh reality is: companies don't care. Yeah, your app might turn out to be a unicorn, but the overwhelming odds are that it won't. And no one cares that you'll tell your other broke friends to avoid the service. We'd all like to think it to be different, that a company might care about appeasing my broke ass. But as already pointed out, they want the whales. I also wonder, despit…

Meanwhile this leaves an opportunity for a different company to provide these services. I do my best to avoid FAANG giants who don't think about me.

The highly price sensitive customer will force you to compete only on price. That's just forcing yourself into a commodity market. It's bad business. I would never try to cater to that market. Very dangerous. Competition will drive margins down to near zero.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#152

Post by jjk166 » Thu, Dec 10, 2020, 7:54 PM UTC

patrickaljord wrote: post #144

jjk166 wrote: post #86

Earlier quoted context omitted.

So you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

> and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong? This is HN, many of us are solo founders with no coworkers or employees. Also how could a "friend" pull the plug? If it was a physical server running in your house maybe, otherwise you can't really give them access to your AWS account with all your private clients data in there.

If you don't have anybody who can monitor your test, and you're not monitoring your test, why are you doing a test?

As for having a non-employee pull the plug, set up an IAM user with permission to access the test instance

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#153

Post by Aperocky » Thu, Dec 10, 2020, 7:56 PM UTC

LordHeini wrote: post #42

That sort of crap is the reason we host all our stuff on root servers. Even trying to read the amazon pricing for their instances, hours and what not, drives me insane. Seems this is done on purpose. no wonder they make so much money with it. So i have never seen a reason to move any stuff to the cloud. Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Its way cheaper, usuall…

> Most apps don't need scaling anyway

This is exactly right. I host stuff in buckets/cloudfront and uses a bit of lambda/route53. I end up paying $4 a month.

now that will be very different if 10 million people suddenly decide to visit my site, but if that happens money probably won't be a problem after all.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#154

Post by raphaelj » Thu, Dec 10, 2020, 7:58 PM UTC

bpodgursky wrote: post #113

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

There are some cloud services where it's not quite this simple. S3 -- you can't just delete customer data because they hit a billing limit RDS -- not going to drop databases on the 27th of the month Anything with persistent data is going to have to stay alive and accumulate costs. Admittedly these services aren't where the crazy bills come from, but it does make a simple kill switch a bit more complex.

You don't have to immediately delete customer data.

Most service that has a limit cap will have a "grace period" of a couple of days during which the service does not work but the data is not deleted. That give your some time to get notified of the issue, and fix the problem/increase the limit.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#155

Post by benlivengood » Thu, Dec 10, 2020, 8:00 PM UTC

a-priori wrote: post #148

ZephyrBlu wrote: post #49

Earlier quoted context omitted.

Probably because it's not so simple on the backend. I'm guessing there's a good chance a lot of systems are only eventually consistent, which could explain why billing takes a long time to update. Aggregation of service usage for billing could also be an expensive operation, so it's only updated irregularly instead of being near real-time. It would be a great feature, but I can imagine it being very complex. It's als…

This is a billing question, not a technical question, and looked at through that lens it's easy to put a hard limit on a monthly bill: just don't ever issue bills greater than that amount. If I say I only want to pay a maximum of $1000 a month, and I hit that limit but it takes a bit for the provider to shut everything down so really $1100 of resources were consumed, then the provider eats the $100 overrun and I get…

It's also a mostly solved problem because advertisers have budgets and it's common to implement globally distributed budget servers to avoid showing more ads than the advertiser paid for, despite tens of thousands of individual web servers needing to know which ads in their inventory have budget left.

It's a fun exercise similar to global rate-limiting/load-balancing.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#156

Post by modeless » Thu, Dec 10, 2020, 8:03 PM UTC

jasonpeacock wrote: post #102

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage. Compute is an active resource, when you exceed your budget it can be automatically shutdown. Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action. Providing fine-grained cost limits help some, as passive reso…

It's not really that complex. All compute should shut down. All API calls should fail. Storage should be (optionally) preserved at normal cost.

Your examples are simple given this framework. Uploading/downloading data to storage is an API call. Monitoring is compute. Metric history storage is storage.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#157

Post by marcell » Thu, Dec 10, 2020, 8:05 PM UTC

If this happens you can usually reach out to Google to see if they will refund the charge. They don’t really benefit from making $72k of a solo developer’s buggy code. I’ve done it once and their team was very helpful and reversed the charge.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#158

Post by raphaelj » Thu, Dec 10, 2020, 8:06 PM UTC

villgax wrote: post #6

I'd happily use a platform that allowed for an option for limiting billing on a daily/monthly/another metric.

Some platforms do that: - Heroku costs are pretty predicable, and you can easily set a maximum scalability threshold to their auto-scalable dynos, so that they will never cost you more than a predefined amount of money; - BunnyCDN requires me to top-up their prepaid account, so that I'll never spend more than what I have on that account.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#159

Post by scrollaway » Thu, Dec 10, 2020, 8:06 PM UTC

csharptwdec19 wrote: post #138

scrollaway wrote: post #136

Earlier quoted context omitted.

The $100K credit (which I've been granted multiple times) is there because if Amazon can get you to invest serious work into their infra, they'll make up for it in the long run. It's not "lock in", it's sales. The only amazon "lock in" really is their bandwidth-out pricing, which is a sleazy tactic for sure but I'm not hesitant to call it out when it's the case . You can get the $100/$300/$1000 tier if you are in "ju…

> Anyway I don't know what your point is, I'm not even sure if you have one. They're not "marketing" their pricing, nor the fact that you are "forced to design systems that use less resources". I think they are referring to this statement: > > AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that…

Overengineering is easy and happens not just with AWS but with just about anything in software engineering.

I'm having trouble seeing how this changes what I'm saying: That with the way AWS pricing is structured, you are supposed to take it into account when designing your product.

When you reach a certain size / complexity and you have to design infrastructure, you should be making schematics, predictions on the usage peaks and troughs, how various parts of the infra will be affected, how active/idle they will be.

When you are dealing with AWS, pricing becomes extremely predictable because it can be derived from those plans. And it is far better to be dealing with that kind of model than to deal with "unlimited with a million asterisks" or something. AWS is predictable, reliable, and most notoriously has never ever increased their prices, so whatever you calculated will not go up because of Amazon's decisions.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#160

Post by jschwartzi » Thu, Dec 10, 2020, 8:07 PM UTC

jjk166 wrote: post #86

onion2k wrote: post #58

Earlier quoted context omitted.

For AWS you get about $0.20 per 1 million requests on Lambda. If you get it right, great. If you get it wrong then you end up doing billions of operations by mistake, which could cost a huge amount. That's what happened to the author of the article. But really the first lesson you should learn in any cloud setup is Billing Alarms Alarms only tell you that something is going wrong. They don't stop it. If your mistake…

So you're taking code that you haven't validated locally to see what resources it uses, you're putting this up on the cloud to test it, then you are immediately going to the middle of nowhere without your laptop/phone/etc, and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong?

If I'm the only developer on a project and I really need to get to market I might do just that. I sometimes do day hikes on weeknights so this is actually a likely scenario for me.