Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

251–260 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#251

Post by stefan_ »

seanwilson wrote: post #173

https://blog.tomilkieway.com/72k-2/ > To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is…

We've all had a program crash from a stack overflow. The problem seems to be that instead of the "serverless panacea" they were promised, the code they built can now only run on one of many Google servers, none of which are theirs. No way to kick the tires at all.

It honestly reminds me of debugging a Jenkins pipeline. Something that was designed to be super generic of a runtime but yet the tooling can inexplicably only live on computers that are not your local development machine, and all of it is maximally painful to stub or test or debug to seduce you into "just running it live".

It's like the opposite of the "small agile team" thing they were talking about. If your program requires 7 API keys and some cloud environment to do a test run, I want no part of it.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#252

Post by alasdair_ »

As a kind of reverse of this, I once had a Google App Engine site with a reasonable $300/day limit on it. We ended up getting a bunch of international news and usage spiked massively (hooray!) and I tried to update the limit manually only to find out I had to wait 24 hours before the new limit applied.

This mean losing a TON of new users :(

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#253

Post by Alupis »

patch_cable wrote: post #249
Alupis wrote: post #246

Earlier quoted context omitted.

Why not alert thresholds, configurable by the user? Email me when we cross $X amount in one day, Text when we cross $Y, and Call when we cross $Z. Additionally, allow the user to configure a hard cut-off limit if they desire. Just provide the mechanisms and allow users to make the call. Google et al would have a much stronger leg to stand on when enforcing delinquent account collections if they provided these mechani…

Something like this for alerts? https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Doesn't look like there's any cutoff mechanism there, and it's a separate, optional step instead of part of the setup flow with a mandatory opt-out warning.

Nor does that address the other complaint - Google (and possibly others) seem to be willing to extend an unlimited credit line to all customers without any prior vetting for ability to pay. That's crazy.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#254

Post by oblio »

shaicoleman wrote: post #245
oblio wrote: post #238

Earlier quoted context omitted.

Well, the real question for all cloud providers, for which I expect crickets as an answer, is: Why don't cloud providers allow setting a budget which cannot be exceeded? A simple, 1-click way to say: this account should never go over $500 a month. Just stop creating resources or responding to requests if it does.

AWS recently released AWS Budget Actions which should allow you to do that. https://aws.amazon.com/blogs/aws-cost-management/get-started...

It still looks rather complex but if it can actually enforce a budget, that would be great.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#256

Post by greatgib »

sudcha wrote: post #186
greatgib wrote: post #69

At the end of page 2, there is a good ass licking bullshit sentence: > He said that as an ex googler and as the beneficiary of a gesture, but this contradict the full history he told us. If the doc and tools were so great, why he felt into this situation?

OP here. Maybe it comes across like that, but as somebody building a product with very limited resources, Google's documentation is one of the best so far. The situation was our fault too. I just went with a test and fail fast attitude, just like with every things we do a dev environment.

Let's be realistic. You are a team of a few persons, using a 'starting' free plan. How great tools and ux could lead you to not have clue on what they are doing and being able to incur a so high cost by surprise?

Also, for example, in which world is it nice to have your service "auto upgraded" from "trial" mode to 72k "full billing" mode without your consent first?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#257

Post by thanksforthe42 »

scrollaway wrote: post #176
thanksforthe42 wrote: post #169

Earlier quoted context omitted.

I think that missed the point. Engineers are applied scientists. Programmers are not applied scientists.

Why does any of this matter?

Different expertise.

I'm guessing you are neither?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#258

Post by dralley »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

>It would mean a catastrophic interruption to the customers application exactly when its the most popular/active.

Making the worst case scenario no worse than traditional infrastructure.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#259

Post by pstuart »

donavanm wrote: post #239
cogman10 wrote: post #206

Earlier quoted context omitted.

I know there's no reason for Google or AWS to do this, but man do I wish there was a way to put down a spending limit and simply disable anything that goes over that limit. It's a little bit nuts that there are no guardrails to prevent you from incurring such huge bills (especially as a solo developer that might just be trying out their services).

The downside of disabling active resources is huge. It would mean a catastrophic interruption to the customers application exactly when its the most popular/active. And theres no practical way to determine whether the customer is “trying it out” or running a key part of their business on any particular resource. On the other hand retroactively forgiving the cost of unexpected/unintentional usage doesnt impact the cus…

There are ways it could be done relatively benignly, such as defaulting to paranoid and explicitly opting out.

And for those that are heading into that financial barrier it should be a straightforward problem to look at trending to anticipate the shutdown and send out an alert.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#260

Post by camjohnson26 »

uoaei wrote: post #162
serial_dev wrote: post #23

Earlier quoted context omitted.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

> It is baffling why cloud providers don't have that option. ...is it? If a lazy dev leaves their corporate account open and you can bill it for their negligence, protected by the contract you already signed, you earn a lot of money. From a purely business perspective, it is stupid(!) to provide a stopgap for that. Edit: to be clear I am not advocating one way or the other. But it is surprising that people are "baffl…

Google is around a trillion dollar company, your $75,000 is a completely immaterial amount to them. Not to mention it would be a one time payment that would drive away customers and lead to bad PR like this post.
Post reply on HN