Burnt $72k testing Firebase and Cloud Run and almost went bankrupt • Hacker News Board

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#101

Post by sofixa » Thu, Dec 10, 2020, 3:42 PM UTC

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Oh please, you can't be serious?

First, that $1200 server costs fixed money upfront, and then you pay per month for colo, and for internet, which usually includes a fixed bandwith cap or limits, with bursts which you pay for. So no, it's not fixed.

Second, a server you have to maintain, harwdare and software-wise, is much more complex, and takes much more time, than a managed service. You want a database? Install it yourself, maintain it yourself, backup it yourself, monitor it yourself. And same with everything else.

Third, there's zero redundancy in your "setup". If you want it with the most basic redundancy, you triple the costs (second server, extra networking equipment, etc.).

Fourth, geo redundancy/distributedness? Please. Good luck if you have someone far away who wants to visit your site.

Fifth, let's say you need to scale. Like, you get 10 more users today than you did yesterday, or you get featured on HN or Reddit or local news or whatever. F. You're looking at months and a lot of cash, upfront.

"A big collection of bare-metal hypervisors" makes sense in some cases, but don't pretend it doesn't come with a non-negligible time spent maintaning it and requires significant upfront capital and man hours to do the same you get easily on a public cloud platform (databases, message brokers, object storage, etc. etc . etc. etc. etc. etc.).

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#102

Post by jasonpeacock » Thu, Dec 10, 2020, 3:50 PM UTC

onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

It's surprisingly complex to do that. Let's take a simple example and say your cloud account is doing 2 things - compute & storage.

Compute is an active resource, when you exceed your budget it can be automatically shutdown.

Storage is a passive resource, when you exceed your budget it can be automatically....deleted? That's almost always the wrong action.

Providing fine-grained cost limits help some, as passive resources usually don't have massive cost spikes while active resources do, so you can better "protect" your passive resources by setting more aggressive cost limits on the active resources.

This quickly gets more complicated. Another example is most monitoring services are a combination of active (actual metric monitoring) and passive (metric history) resources. A cost limit on that monitoring service likely won't provide sub-service granularity, mostly depending on whether the service even has different charges for monitoring vs history.

Oh, also, even for a passive resource like storage, you also have active resource charges whenever you upload/download your data.

Ugh, what a mess. The best thing to do is pay attention to your spending, just like you do with your personal & corporate budget.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#103

Post by dilatedmind » Thu, Dec 10, 2020, 4:08 PM UTC

interesting, how did the spend breakdown between cloud run and firebase?

did you have any limit to how many req/s you made to an individual site? It seems this would be difficult to implement with this architecture.

how did you deal with following links in circles/ avoiding scraping the same page multiple times?

I had built something similiar at a previous job, recursively scraping ecommerce sites. The first thing I noticed was some of the sites we were scaping couldn't handle more than a couple requests a second (in particular as we scaped uncached pages by sites running php). Other sites were quick to ip ban.

I kept things simple, a few dozen micro instances on aws (think they were like $3 a day) running puppeteer. A single server acting as a controller, keeping a per site queue and allowing us to set per site request limits if necessary. All the state of which links were already seen just kept in memory. Of course everything was also persisted to a db, and if the controller process needed to be restarted, it could restore the queue/ seen state and resume.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#104

Post by walrus01 » Thu, Dec 10, 2020, 4:15 PM UTC

sofixa wrote: post #101

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Oh please, you can't be serious? First, that $1200 server costs fixed money upfront, and then you pay per month for colo, and for internet, which usually includes a fixed bandwith cap or limits, with bursts which you pay for. So no, it's not fixed. Second, a server you have to maintain, harwdare and software-wise, is much more complex, and takes much more time, than a managed service. You want a database? Install it…

yes, I am serious, because as described in the original post this was somebody's test/prototype environment. Which is the ideal use case for a DIY scenario, until you're ready to send things into production.

I have seen people spend thousands of dollars on a cloud hosting platform to develop and test something when it could have been done equally well on a 4-year-old desktop PC sitting on somebody's desk. If they had only thought to bother installing the same (debian, centos, whatever) environment + packages + custom configuration on it.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#105

Post by sofixa » Thu, Dec 10, 2020, 4:50 PM UTC

jcelerier wrote: post #53

LeonM wrote: post #39

Earlier quoted context omitted.

You confuse cost with salary. Someone who makes 35k a year costs much more. Think about office space, training, insurance, payroll costs etc.

35k is the gross which encompasses a big part of that except office space.

No it's not. In France companies pay ~1.5-2x gross salary in total, gross going to the employee ( some of it getting deducted by the state), rest going for health insurance, taxes, etc.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#106

Post by FastQ » Thu, Dec 10, 2020, 4:55 PM UTC

marktolson wrote: post #68

FastQ wrote: post #40

I'm just a student but I've spent about 10 hours trying to figure out why Azure has been charging me >$5/day for their "basic" database @5DTUs, 2gb max storage. This morning I was so exasperated I sent a letter threatening to report them for fraud if nobody could tell me why I was being charged 30x the listed rate, which so far no one has. This is an extremely cathartic post to see that I'm not alone, thanks for shar…

Go to billing > cost analysis > filter by resource break down. Azure billing analysis is pretty amazing.

Yeah, but it just shows my database cost which is higher than is listed as far as I can tell.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#107

Post by steren » Thu, Dec 10, 2020, 5:00 PM UTC

YawningAngel wrote: post #34

tlarkworthy wrote: post #12

And that's why you set the 'autoscaling.knative.dev/maxScale' for cloud run https://github.com/futurice/terraform-examples/blob/2ccb2fa3...

A sane implementation would default this to a low value

Cloud Run PM here: I'm sorry for the bad experience the customer shared in this article, we could certainly do better with bill management.

We pick 1,000 as a default value for "maxScale", this can be considered high for some users, but low for users who expect infinite scaling from the service and start with a load test to evaluate it.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#108

Post by nojito » Thu, Dec 10, 2020, 5:14 PM UTC

scrollaway wrote: post #70

nojito wrote: post #67

Earlier quoted context omitted.

AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.

The margin is absolutely not the same across all products. > There is a massive secondary consulting market because of AWS's price obscurities. Its. Not. For. You. AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill. That consulting market is…

>AWS pricing is a part of your design. With some exceptions (that you aren't talking about), they charge you more for using more resources. You are forced to design systems that use less resources if you want to optimize your bill.

This is marketing.

It's like saying you want to build a house and the quote you got ends up blowing up 100x overnight.

Great example is the 100k credit for startups. You can repeat it's not for you all you want, but their business is predicated on pricing ignorance and vendor lockin.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#109

Post by [deleted] » Thu, Dec 10, 2020, 5:22 PM UTC

klohto wrote: post #75

nojito wrote: post #67

Earlier quoted context omitted.

AWS's margin is currently 30+% which is massive. >AWS pricing is not obscure There is a massive secondary consulting market because of AWS's price obscurities.

> There is a massive secondary consulting market because of AWS's price obscurities. While that's true, there is consulting market for most things that are complicated. Doesn't mean they are shady. It's simply not for you. You are welcome either to dive in or get a consultant. I promise you though, that AWS pricing isn't difficult once you understand few concepts and know your way around the Cost Explorer. With prope…

[deleted]

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#110

Post by dang » Thu, Dec 10, 2020, 6:15 PM UTC

walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

Please don't be a jerk on HN, especially in response to someone else's misfortune, even if they brought it on themselves. Maybe you don't need to treat these people better (though why not?) but you owe the community better if you're posting here. If you wouldn't mind reviewing the site guidelines and taking the intended spirit to heart, we'd be grateful. Note these ones: "Be kind" and "Please don't sneer"

https://news.ycombinator.com/newsguidelines.html

p.s. I skimmed through your recent commenting history and it looks great—just the kind of thing we want here. Sharing some of what you know is exactly what we want users to do. But please don't be supercilious about it, as in this comment and https://news.ycombinator.com/item?id=25372847. Ignorance doesn't deserve humiliation, and that ingredient poisons the ecosystem (and eventually starts a degrading spiral, e.g. https://news.ycombinator.com/item?id=25373520). The rest is good.