Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

171–180 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#171

Post by WrtCdEvrydy »

jjk166 wrote: post #152
patrickaljord wrote: post #144

Earlier quoted context omitted.

> and you can't arrange for a coworker or friend to pull the plug for you if something goes wrong? This is HN, many of us are solo founders with no coworkers or employees. Also how could a "friend" pull the plug? If it was a physical server running in your house maybe, otherwise you can't really give them access to your AWS account with all your private clients data in there.

If you don't have anybody who can monitor your test, and you're not monitoring your test, why are you doing a test? As for having a non-employee pull the plug, set up an IAM user with permission to access the test instance

> you're not monitoring your test, why are you doing a test?

Agile. Bringing you bankruptcy at the speed of cloud.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#172

Post by WrtCdEvrydy »

throwaway201103 wrote: post #168
that_guy_iain wrote: post #50

Earlier quoted context omitted.

> That sort of crap is the reason we host all our stuff on root servers. Having just started my own journey into building products for myself, pretty much the first thing I realised with my tech was I need to get dedicated servers instead of cloud just because it costs 100x less. > Just grab a dedicated server for a few bucks and put a bunch of docker containers on those. Exactly, if you really want kubernetes coolne…

> kubernetes ... is super easy to setup Can you point me to the super easy setup guide? Because I've tried a few and never gotten it working.

I don't like use kubernetes raw but I am a fan of Caprover (which has kubernetes support)

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#173

Post by seanwilson »

https://blog.tomilkieway.com/72k-2/

> To overcome the timeout limitation, I suggested using POST requests (with URL as data) to send jobs to an instance, and use multiple instances in parallel instead of using one instance serially. Because each instance in Cloud Run would only be scraping one page, it would never time out, process all pages in parallel (scale), and also be highly optimized because Cloud Run usage is accurate to milliseconds.

> If you look closely, the flow is missing few important pieces.

> Exponential Recursion without Break: The instances wouldn’t know when to break, as there was no break statement.

> The POST requests could be of the same URLs. If there’s a back link to the previous page, the Cloud Run service will be stuck in infinite recursion, but what’s worst is, that this recursion is multiplying exponentially (our max instances were set to 1000!)

Did you not consider how to stop this blowing up before implementing? Having one cloud function trigger another like this with no way to control how many functions are running at the same time with no simple and quickly met termination condition (with uncapped billing) is playing with fire. It's not going to be optimal either if most of the time each function is waiting for the URL data to download.

You need to be using something like a work queue, or just keep life simple and keep it on a single server if you can.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#174

Post by kabirgoel »

Alupis wrote: post #165

As J. Paul Getty once mused[1]: > If you owe the bank $100 that's your problem. If you owe the bank $100 million, that's the bank's problem. Crappy situation for OP and his startup, but I find the part about reading up on bankruptcy to be a bit premature. Perhaps not the most ethical choice, but what stops OP from just not paying the bill, and finding a different cloud provider? Obviously they'll want to not repeat t…

One of my favorite quotes of all time. J. Paul Getty was quite the weirdo. His Wikipedia article is worth a look, especially the section on his frugality.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#175

Post by abawany »

greatgib wrote: post #64
serial_dev wrote: post #23

Earlier quoted context omitted.

It is baffling why cloud providers don't have that option. I might want to have an app because I don't mind spending 50 dollars on my pet project as a hobby, but I don't ever want to spend more than that. Not if I write a wrong query that's suddenly becomes very expensive, not when I got attacked, and not even when I have legit users. By the way, the same goes for some companies, too, just the threshold would be diff…

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

Every time a GCloud rep would ask us about what we need, we would say: fix the billing interface. As far as I know, it never got fixed. The feelings I would get when looking at cloud billing interfaces can be summed as: obfuscated, like a pawnshop, and caveat emptor. I kind of came to the conclusion that if the cloud giants are not fixing their billing interfaces, then just like Amazon not sending you the details of the items you ordered by email and thus causing you to use the app to help with primenesia, there is a 'business' reason why the billing interfaces are generally incomprehensible.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#176

Post by scrollaway »

thanksforthe42 wrote: post #169
csharptwdec19 wrote: post #137

Earlier quoted context omitted.

The shift from 'Developer/Programmer' to engineer has indeed been part of a push away from creativity towards cookie-cutter work. An interesting analogue would be the Automotive industry; As time progressed, Companies focused more and more on 'engineering' versus art/tradition/etc. But as the industry evolved, "Flashy" vehicles that took risks became moreso either a halo product for a brand, or relegated to Luxury/Bo…

I think that missed the point. Engineers are applied scientists. Programmers are not applied scientists.

Why does any of this matter?

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#177

Post by WrtCdEvrydy »

csharptwdec19 wrote: post #138
scrollaway wrote: post #136

Earlier quoted context omitted.

The $100K credit (which I've been granted multiple times) is there because if Amazon can get you to invest serious work into their infra, they'll make up for it in the long run. It's not "lock in", it's sales. The only amazon "lock in" really is their bandwidth-out pricing, which is a sleazy tactic for sure but I'm not hesitant to call it out when it's the case . You can get the $100/$300/$1000 tier if you are in "ju…

> Anyway I don't know what your point is, I'm not even sure if you have one. They're not "marketing" their pricing, nor the fact that you are "forced to design systems that use less resources". I think they are referring to this statement: > > AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that…

> Suggested a very overdone architectural revamp, moving everything possible into AWS Specific technologies.

To be honest, depending on the technology, the savings could be worth it... for example, did you know you get a discount if your traffic is served over cloudfrount? even if your distribution is set to no cache any resource, you can front your APIs using cloudfront and save networking.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#178

Post by quesera »

steren wrote: post #107
YawningAngel wrote: post #34

Earlier quoted context omitted.

A sane implementation would default this to a low value

Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management. We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.

> We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.

That seems absurd to me.

I think it makes much more sense to put the onus on the sophisticated customer to increase their maxScale to an unusual value. Users who "expect infinite scaling...and start with a load test" are sophisticated users.

E.g. set maxScale low, like 2 or 4. The sophisticated customer would recognize their oversight quickly. Click-click, fixed, restart test.

Effectively 100% of less-sophisticated customers will not need enormous scale on day 1. Customers with whom you do not have an existing billing relationship in the 10s of thousands of dollars per cycle will almost certainly not want it.

I'd consider that level of overspecification to be a strong anti-pattern.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#179

Post by beoberha »

ctvo wrote: post #146
greatgib wrote: post #64

Earlier quoted context omitted.

It's not complicated to add configurable hard limits for these companies but they don't allow it because the current situation is more interesting for them. They want to suck the maximum money from consumers before they realize. For one person that will complain wildly and having to do a gesture, there are hundreds other companies that will not notice or just pay without recourse.

> They want to suck the maximum money from consumers before they realize. This is a naive understanding of how corporations like Google and Amazon work. Bad will and using gym membership tactics aren't how they scale or make money. Getting you to confidently try things knowing you won't get charged (the reason they have those free tiers) so you'll get your company, your start-up, your next side project on it is much…

I work in this space and you’re absolutely correct. Your last paragraph hits the nail on the head for pretty much every complain people have about the public clouds.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#180

Post by WrtCdEvrydy »

mlboss wrote: post #94

I don't understand why developers use cloud for bootstrapping/side project. Digital Ocean is all you need $5 droplet + $15 Postgres or even better $7 dyno on Heroku.

$5 droplet, 2GB swap (set it internally) and run Caprover.

Deploy your Postgres (for DB), minio (for s3 storage) and your webapp from Caprover. Add nodes as you need to scale out.

Post reply on HN