Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

31–40 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#31

Post by dang »

vuln wrote: post #15
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

You only have to be 'terrified' if you don't write good code. Maybe people should actually test their code before pushing it to production? It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '

Please don't be rude.

https://news.ycombinator.com/newsguidelines.html

Re: How we spent $30k in Firebase in less than 72 hours

#33

Post by mirimir »

jsmthrowaway wrote: post #28
mirimir wrote: post #5

But wait, the image shows ~$600 million. So $30K is small. Maybe that ~$600 million isn't in USD?

No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million. The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a…

No, it shows "$600,043,603". Where do you see "mil"?

But OK, I didn't get that "COP" is Colombian Peso :(

And there's another image that shows total collected as "USD $244,875". The ratio is 2450, which is close enough to the exchange rate.

Re: How we spent $30k in Firebase in less than 72 hours

#34

Post by JoJoJoboop »

vuln wrote: post #15
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

You only have to be 'terrified' if you don't write good code. Maybe people should actually test their code before pushing it to production? It's not firebase's fault you ran bad code and have a huge bill. They have bills to pay also. Think they can tell their vendors 'sorry we can pay you this week. A customer ran up a 30k bill and they can't pay it. So we can't pay you right now. but lol bad code right? '

Well, since Firebase has lowered the barrier of entry with their easy setups, I think it is justified to expect at least some developers to make mistakes.

I don't blame Firebase at all though -- great product.

Re: How we spent $30k in Firebase in less than 72 hours

#35

Post by jsmthrowaway »

mirimir wrote: post #33
jsmthrowaway wrote: post #28

Earlier quoted context omitted.

No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million. The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a…

No, it shows "$600,043,603". Where do you see "mil"? But OK, I didn't get that "COP" is Colombian Peso :( And there's another image that shows total collected as "USD $244,875". The ratio is 2450, which is close enough to the exchange rate.

The Y axis of the graph which actually has relevant information. You are looking at a campaign page, and assuming those Colombian pesos are available to the author’s team.

When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.

Re: How we spent $30k in Firebase in less than 72 hours

#36

Post by ghaff »

thieving_magpie wrote: post #10
ghaff wrote: post #7

Earlier quoted context omitted.

From what I understand, AWS at least is pretty good about giving you at least one "get out of jail free" card if things go awry. Caps are tough though. I can certainly understand a use case that would want a hard circuit breaker that just kills everything it can once it hits a certain threshold. Sort of; you presumably don't want everything on S3, for example, to be deleted. On the other hand, moving up the scale of…

There's nothing guaranteed about a "get out of jail free" card, so factoring that into your decision making is probably a bad road to go down.

I'm not suggesting that. For a lot of casual users I'm not sure that cloud services are necessarily the best approach for an externally-facing service. They may well be better with a VPS that has a hard resources/dollar limit.

Perhaps cloud providers should have some sort of hard circuit breaker option (though it won't help for some things like storage) but it's probably not a priority as not a lot of businesses--their primary customers--would be OK with effectively hitting the power button for their entire cloud account if they exceed some dollar amount that someone or other configured a couple of years ago.

Re: How we spent $30k in Firebase in less than 72 hours

#37

Post by sciurus »

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem.

Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws.

Re: How we spent $30k in Firebase in less than 72 hours

#38

Post by mirimir »

jsmthrowaway wrote: post #35
mirimir wrote: post #33

Earlier quoted context omitted.

No, it shows "$600,043,603". Where do you see "mil"? But OK, I didn't get that "COP" is Colombian Peso :( And there's another image that shows total collected as "USD $244,875". The ratio is 2450, which is close enough to the exchange rate.

The Y axis of the graph which actually has relevant information. You are looking at a campaign page, and assuming those Colombian pesos are available to the author’s team. When you said “the image” I thought you were looking at the right one, and I thought it odd you were off several orders of magnitude from what I assumed to be your misunderstanding. That explains that. I had to go back and find your figure.

Ah, I didn't look closely at the charges graph. So yes, it maxed at ~5000 USD per day.

I gotta say that using "$" for both USD and COP is confusing. So you must say "USD $x" and "COP $x". Then why bother with the "$"?

Re: How we spent $30k in Firebase in less than 72 hours

#39

Post by ithkuil »

Something1234 wrote: post #27
marktangotango wrote: post #17

Earlier quoted context omitted.

What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?

One bankrupts you, the other just has a minor hiccup with a few lost customers that day.

a (cost) cap is a feature designed to prevent you from going bankrupt but at the expense of not recovering from the surge after you deplete the cap until the end of the cap granularity.

So, if the cost cap is defined as $n per day, once you deplete it you'll be down for the rest of the day (or if you take some manual action to increase the cap, if the cloud provider supports it).

This problem is a function of the granularity. Imagine a system that let you say:

"I want to spend max $n per second with a extra burst of $m per day/week"

You adjust your "$n" to match the throughput you'd have if you had a fixed size "pay what you provision" system and reserve $m for lucky events as landing on HN.

The amount of planning you have to do is similar to the traditional resource allocation, but with the benefit of paying less than provisioned if you're not getting all that traffic.

Re: How we spent $30k in Firebase in less than 72 hours

#40

Post by sitkack »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

> Reason through your system

modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this?

I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an affordance for application maintenance and visibility. Differential analysis and graphing built directly into the execution substrate.

Judge away!

Post reply on HN