Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

blog.tomilkieway.com

Post reply on HN
Print view View original

281–290 of 397 posts

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#281

Post by mmcconnell1618 »

ylere wrote: post #145
luser007 wrote: post #132

Earlier quoted context omitted.

Could it be listed "hourly" and you're charged "daily"? Add in VAT (equal to 25% in some countries) and you match the 30 times higher than expected charge.

https://azure.microsoft.com/en-us/pricing/details/sql-databa... Basic tier, 5 DTUs, 2 GB is listed as ~$4.8971/month or $0.0068/hour on this page. Extra storage would cost more but is not available for the basic tier.

Do you have geo-replication turned on? More regions will be an additional $5/month (plus bandwidth between regions) if you replicate. You can serve everything out of a single region but it is pretty easy to add others if you're not paying attention during initial setup.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#282

Post by dathinab »

donmcronald wrote: post #51
onion2k wrote: post #3

The fact that cloud providers don't have a simple "This is how much I can afford, don't ever bill me more than that!" box on their platforms makes development a lot scarier than it really needs to be.

This is my worst nightmare. Lol. I guess now is a great time to give Azure a shoutout for sitting on their hands for 8 years without so much as a response to the community for half a decade [1]. At least AWS allows using a prepaid credit card so they’ll need to call me if things go haywire. I bet if that $72k charge went through it would have been much harder to get out of. “Sorry, we don’t have the money” is a much…

But then consider following hypothetical but possible scenario:

Sorry until you pay, no more Amazone services for your company.

Now you must move to a new cloud provider (or make a new company).

Oh, wait they now interchange (bad) customer information to better find fraud and you just got marked at "owning a lot to amazon" so no cloud for you anymore at any provider.

Now you want to buy your own hardware. So you need a credit from the bank, but dang, your owe to much to a big company and the bank now, so no credit for your company either.

While part of above's scenario is luckily not how reality currently works. But then who knows when (part of) such a horror scenario becomes reality.

In the end relaying on forcibly not paying back money you contractually own is just not a very viable strategy in my view.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#283

Post by mmcconnell1618 »

altdatathrow wrote: post #167

Do not use hosted cloud services where the implementation creates publicly accessible API keys and each HTTP request results in a charge to your account. A few specific examples are Firebase, Algolia, and AWS Lambda. All it takes is one programming mistake or one bad actor and you can find yourself in an equally precarious situation.

Yep, you should wrap then in something like Apigee or an API Server that can throttle requests and keep traffic to reasonable numbers for your service.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#284

Post by dathinab »

ggthrowaway2020 wrote: post #194
sandGorgon wrote: post #112

Earlier quoted context omitted.

Google does have this feature https://cloud.google.com/billing/docs/how-to/budgets-program... Here's the specific example https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

As a former victim to the same issue as OP, I am furious every time I see a Googler promote that as a solution. In our case, we racked up a $10000 bill on BigQuery in ~6 hours, when a job was failing and auto-retrying. We had set up every alert correctly and our reaction time was about 5 minutes (about $100 of usage, no big deal). So how did we get a $5000 bill? Google's alert was 6 hours late (according to them, thi…

(By now) GCP documentation says billing alerts can be late by days. Yes days not just hours. Totally crazy.

EDIT:

Link (see blue info box a a bit below the anchor on which the page is opened):

https://cloud.google.com/billing/docs/how-to/notify#cap_disa...

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#285

Post by 1337shadow »

salmonlogs wrote: post #207
walrus01 wrote: post #13

writing as somebody who runs a big collection of bare-metal hypervisors for ISP infrastructure purposes... this post quite honestly just makes me smirk. I have truly lost track of the numerous instances, and number of people who would be better served by buying a $1200 test/development 1U dual socket server with a few fast SSDs in it, and putting it in colocation somewhere for a few hundred dollars a month. The costs…

And this comment makes you look incredibly naive and narrow minded. Running some code on a CPU != running a startup. Great you can buy a Dell server on eBay, or you can build a powerful desktop, or rent a VM or get a droplet or scrape on lowendbox. These are not a secret and there is a great reason no one does this other than hobbyists and neckbeards. You do your testing and it works, then what? You have to deliver s…

Really, "a million other supporting services and all the systems that come with it" ?

You get a server with SSH, then you need something expose your container stacks over HTTPS like Traefik (which auto configures), and something for alerting such as Netdata (which auto configures too!), both of which are just a single binary to configure and setup, and probably it won't take long before you have scripts to automate that like we do[0]

Not only do you get amazing prices[1] but ...

You get to be part of an amazing community running the world on Free Software

But yeah, maybe I'm "missing the bigger picture" by "not locking myself in proprietary frameworks"

[0] https://blog.yourlabs.org/posts/2020-02-08-bigsudo-extreme-d... [1] https://oneprovider.com/

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#286

Post by vincentmarle »

salmonlogs wrote: post #195

As an ex-Googler working in a customer facing role in Cloud you did very well to get a $72k bill written off! It's definitely possible but requires a lot of approvals and pulling in a few favours. I went through the process to write off a ~$50k bill for one of my customers and it required action every day for 3 months of my life. Whoever helped you inside Google will have gone to a LOT of trouble, opened a bunch of t…

I went through this very scary experience recently as well (although in our case it was $17K, not $72K). One of our devs accidentally caused an infinite loop with one of the Google Maps paid APIs on our dev account and within hours both our prod and dev accounts were suspended (pro tip: don't link your prod account to the billing account of your dev account). The worst part was that after removing the suspension, our cloud functions were broken and had to be investigated and fixed by Google engineers themselves resulting in our prod app being down for 24 hours... be very careful.

Luckily we were able to get $11K refunded on our card and received $6K credits after spending all night with Google support.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#287

Post by dsymonds »

Exuma wrote: post #270
sudcha wrote: post #267

Earlier quoted context omitted.

OP here. Your assumption is incorrect. I haven't been in touch with anyone in Google, and used 0 internal connections. Happy to make another post with my conversation + documentation to support this. I reached out to the GCP through their regular channels. This is not a paid post, and we are not sponsored by Google in anyway.

You only wrote half the story and no technical details...... why even write a post at all, it's just clickbait and you give no other information how what went wrong, how it got fixed, what exact technical problem was...

Did you read Part 2, linked from the bottom? It has plenty of technical details.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#288

Post by 1337shadow »

Axsuul wrote: post #213
senko wrote: post #44

Earlier quoted context omitted.

> I'm not sure that "colo is cheaper" is one. It absolutely is (cheaper, and a good argument). As an example: we're in the process of switching from Digital Ocean to Hetzner for a project, that will increase infrastructure performance (roughly memory/cpu/storage) by 4x and decrease costs by 4x. And no driving to the colo center is neccessary, as it's their dedicated server, so their on-site engineers do the hardware…

Can anyone recommend a similar provider but for North America?

Sure thing https://oneprovider.com/dedicated-servers-in-north-america

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#289

Post by mmcconnell1618 »

walrus01 wrote: post #104
sofixa wrote: post #101

Earlier quoted context omitted.

Oh please, you can't be serious? First, that $1200 server costs fixed money upfront, and then you pay per month for colo, and for internet, which usually includes a fixed bandwith cap or limits, with bursts which you pay for. So no, it's not fixed. Second, a server you have to maintain, harwdare and software-wise, is much more complex, and takes much more time, than a managed service. You want a database? Install it…

yes, I am serious, because as described in the original post this was somebody's test/prototype environment . Which is the ideal use case for a DIY scenario, until you're ready to send things into production. I have seen people spend thousands of dollars on a cloud hosting platform to develop and test something when it could have been done equally well on a 4-year-old desktop PC sitting on somebody's desk. If they ha…

But one big benefit of cloud providers is that you can spin up those servers for testing and when it doesn't work out as expected, you can just magically make them go away. If you're putting out the capital to buy servers in a rack, you have to use them all the time for the cost benefit to work out. A regular test environment that is used all the time? Yes, that could possibly be cheaper if you purchase the hardware but you also need to amortize the cost of purchasing new servers every 3 to 5 years to get the equivalent of the equipment provided by the cloud provider.

Re: Burnt $72k testing Firebase and Cloud Run and almost went bankrupt

#290

Post by aidenn0 »

dsymonds wrote: post #287
Exuma wrote: post #270

Earlier quoted context omitted.

You only wrote half the story and no technical details...... why even write a post at all, it's just clickbait and you give no other information how what went wrong, how it got fixed, what exact technical problem was...

Did you read Part 2, linked from the bottom? It has plenty of technical details.

I didn't realize that was a link; seriously why does every website have to make links look different.
Post reply on HN