Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How not to get a $30k bill from Firebase

medium.com

Post reply on HN
Print view View original

81–90 of 114 posts

Re: How not to get a $30k bill from Firebase

#81

Post by arebours »

cryptica wrote: post #30
paxys wrote: post #22

Billing problems aside, it's amazing that web development has reached a stage where it's almost too easy to scale a site to 2m active sessions and 20m+ page views in two days with just off-the-shelf tools and barely any specialized skills.

Except for very simple use cases, serverless platforms in general encourage bad software design patterns. For example, in this case, if a regular database was used instead of Firebase, "counting the number of supporters" would have been done inside the database using a query. The worst that the developer could have done is forget to index to the relevant column; performance would have been sub-par but still orders of…

> Except for very simple use cases, serverless platforms in general encourage bad software design patterns.

> For example, in this case, if a regular database was used instead of Firebase, "counting the number of supporters" would have been done inside the database using a query.

I don't think this is connected to serverless achitecture. You have to write a database query either way. And it would be just as easy to count or retrieve entities one by one with traditional approach.

Re: How not to get a $30k bill from Firebase

#82

Post by inlined »

cryptica wrote: post #30
paxys wrote: post #22

Billing problems aside, it's amazing that web development has reached a stage where it's almost too easy to scale a site to 2m active sessions and 20m+ page views in two days with just off-the-shelf tools and barely any specialized skills.

Except for very simple use cases, serverless platforms in general encourage bad software design patterns. For example, in this case, if a regular database was used instead of Firebase, "counting the number of supporters" would have been done inside the database using a query. The worst that the developer could have done is forget to index to the relevant column; performance would have been sub-par but still orders of…

Not all databases support counting indexes. This is why counts we’re a bit issue at Parse: mongo didn’t support them and if the count is for an unindexed query, it could be a full table scan on a multi-tenant system. Just as bad, mongo as of 3.0 (last I worked with it heavily) did not ever yield during an index scan so counting large numbers on an indexed query could cause massive problems too.

I recommend the atomic increment operator for counts that both Parse and Firestore offer for this problem.

Re: How not to get a $30k bill from Firebase

#83

Post by GFischer »

donmcronald wrote: post #58
dvfjsdhgfv wrote: post #48

Earlier quoted context omitted.

Only Amazon is adamant at refusing this, Microsoft already made it possible some years ago, and I believe Google too.

Azure doesn’t have it for pay as you go. https://feedback.azure.com/forums/906772-cost-management/sug... A 6+ year wait for such a reasonable feature sucks.

And they have it for almost all other plans, so the technology is there.

It sounds like an intentional product decision and not a technical obstacle.

Re: How not to get a $30k bill from Firebase

#84

Post by inlined »

EugeneOZ wrote: post #62
stickfigure wrote: post #32

Earlier quoted context omitted.

You can? I have daily limits for my Google App Engine apps.

Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.

If the concern is DB operations adding up you can lower your quota as well which will be strongly enforced and shut down your app.

Re: How not to get a $30k bill from Firebase

#85

Post by inlined »

stickfigure wrote: post #32
paulgb wrote: post #23

I really wish cloud providers would allow users to set a hard budget that just stops the service if you exceed a threshold. I got a surprise bill from AWS this month (fortunately orders of magnitude less than this, but still ~15x my usual) and am thinking of moving from Lambda to a VPS just so the possibility of this doesn't keep me up at night.

You can? I have daily limits for my Google App Engine apps.

Firebase also has a fixed price plan that can shut off your app if it blows through quotas. It’s called “Flame”

Re: How not to get a $30k bill from Firebase

#86

Post by cc81 »

intricatedetail wrote: post #55
StreamBright wrote: post #52

Earlier quoted context omitted.

Interesting, I saved roughly 2M USD / year for customers moving them to the cloud. Your example is silly, all of the CDN I set up has very strict limit on how much a single IP can use and have multiple alerts if you are passing 100, 200, 500, etc. USD limits. On the top of that if that is not enough you can add more limitations to avoid that exact scenario that a public resource can be abused to cause you financial t…

You see this is too complicated and "If you do it right" imply you can get it wrong and go bankrupt. I prefer to use dedicated server with dedicated bandwidth and Kubernetes on top of that. This makes me sleep comfortably at night. I tried the cloud and this was just too much anxiety for me to handle.

It is absolutely not complicated compared to running Kubernetes yourself. You don't have to use serverless and infinite scaling. For things like Azure App Services you don't autoscale to start with and if you want autoscaling it looks like this:

https://docs.microsoft.com/en-us/azure/azure-monitor/platfor...

You won't get many surprises there.

Re: How not to get a $30k bill from Firebase

#87

Post by cc81 »

BartBoch wrote: post #54
StreamBright wrote: post #52

Earlier quoted context omitted.

Interesting, I saved roughly 2M USD / year for customers moving them to the cloud. Your example is silly, all of the CDN I set up has very strict limit on how much a single IP can use and have multiple alerts if you are passing 100, 200, 500, etc. USD limits. On the top of that if that is not enough you can add more limitations to avoid that exact scenario that a public resource can be abused to cause you financial t…

Not taking sides here, but I think that the cloud is complicated enough (especially for a person that does not specialize in it), to miss one of the edge cases that can lead to the huge bill. In a few minutes, I can set simple PHP script with curl, that will launch 100 requests each second, using a pool of hundreds of thousands of IP's thank to the rotating VPN. This is an edge case of course, but it can happen. I us…

There is a difference between "cloud" and serverless though. I bet most people are not using something that can autoscale infinite by default.

Re: How not to get a $30k bill from Firebase

#88

Post by nestorherre »

laken wrote: post #75

I have to mention this every time I see it... The crowdfunding campaign mentioned in this article took place in "Colombia" not "Columbia." There is no country by the name of "Columbia." This problem is so aggravating to Colombians they even sell t-shirts in their airports that say "IT'S COLOMBIA NOT COLUMBIA"

Why is this getting downvoted? As a colombian myself I find it frustrating everytime it's spelled wrong. Upvoted.

Re: How not to get a $30k bill from Firebase

#89

Post by asciimike »

EugeneOZ wrote: post #62
stickfigure wrote: post #32

Earlier quoted context omitted.

You can? I have daily limits for my Google App Engine apps.

Don't forget they are only warnings, they will not shut down your app and their update rate is not as frequent as you might expect.

App Engine PM: the GAE Daily Spend limits are hard caps that will shut down services when hit. They are different from the broader GCP Billing Alerts which are simply notifications.

The issue is that they only apply to certain GAE services (compute, legacy APIs, etc.) and not across the platform.

Re: How not to get a $30k bill from Firebase

#90

Post by asciimike »

genericacct wrote: post #27
sessy wrote: post #2

TLDR: The huge bill was a result of an improper way the application was coded. They contacted Google/Firebase who were gracious enough to waive off the bill.

That's very nice of them. However I'd have to read all of firebase's small print before i would consider using it. Does their SLA guarantee data availability even if google dcide to spin it off or sunset the way they did with google+, reader and such?

Former Firebase PM: firebase.google.com/terms has all the details.

TL;DR: minimum 1 year deprecation policy, and you've got access to the data at any time (e.g. do a backup and get everything as JSON, or download an entire store bucket and transfer it to S3).

Post reply on HN