Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

21–30 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#21

Post by paranoidrobot »

iamgopal wrote: post #16
alasdair_ wrote: post #14

Earlier quoted context omitted.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

So basically we want to have cake and eat it too.

This seems like an overly cynical/snarky response.

It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.

Re: How we spent $30k in Firebase in less than 72 hours

#23

Post by ShaneOG »

itcmcgrath wrote: post #11
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

A current happy Google Cloud customer here, who also received the Startup credits. When you have credits on your account none of the alerts or budgets work (at least they didn't when I was using it). The only thing you could do is look at your figures daily and plan/scale based on the running total.

Re: How we spent $30k in Firebase in less than 72 hours

#24

Post by benguild »

alasdair_ wrote: post #14
itcmcgrath wrote: post #11

Earlier quoted context omitted.

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

Yeah, that would happen with the Google Maps API before. 150k request limit was not enough for a front-paging, and there was no rollover.

Re: How we spent $30k in Firebase in less than 72 hours

#25

Post by itcmcgrath »

shaunpersad wrote: post #6

I can't say enough good things about Firebase and GCP in general, but I'm always cautious when using Firestore in particular. I usually avoid unbounded queries altogether, and treat it primarily as a key/value store to get by id. When I do use queries, it's always in places where the results have a well-defined limit (usually limit = 1), e.g. finding the most recent X or the highest X. With the above two, you get all…

We've also been rolling out updates to rules to enable you to enforce these types of things. There is performance implications to limit queries for the real-time update system at scale, but for most use cases this shouldn't be a problem.

Definitely more we can improve here for control, and we're open to feedback.

Re: How we spent $30k in Firebase in less than 72 hours

#26

Post by riquito »

> we were using Angular V.4 and we decided to upgrade everything to V.6. It was a huge risk and we wanted this campaign to be perfect, so we did it! In just a day and a half, our team had the first release ready in the new version. After some tests, it looked like the refactor helped the app’s speed, but it was not as fast as we wanted it. Our goal was to load in 3 seconds and it wasn’t working as we expected.

You want it perfect and you cannot afford to put down the site but you're willing to take a "huge risk" based on (wrong) guesses, with clearly not enough time for proper QA. I sincerely suggest you to slow down and reflect on priorities and risk assessment, there's a reason if that firebase code slipped through. By the way I'm happy you avoided the worst case scenario, good luck with your project

Re: How we spent $30k in Firebase in less than 72 hours

#27

Post by Something1234 »

marktangotango wrote: post #17
everdev wrote: post #4

Earlier quoted context omitted.

Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.

What’s the difference between a cap limiting your traffic and server cpu maxed out, or db connection pool or ... limiting your traffic?

One bankrupts you, the other just has a minor hiccup with a few lost customers that day.

Re: How we spent $30k in Firebase in less than 72 hours

#28

Post by jsmthrowaway »

mirimir wrote: post #5

But wait, the image shows ~$600 million. So $30K is small. Maybe that ~$600 million isn't in USD?

No, it doesn’t. In many other locales “mil” means thousand, unlike slang for million. This is why in finance, $5mm means five million dollars. Five mil mil. Five thousand thousand. Five million.

The graph shows a spike to around $5,000 per day ($5 mil por día). The entire dashboard is in USD, presented in a Spanish locale. That is also why the dollar sign is suffixed, the months are not capitalized, and why May has a dot after it, because it is abbreviated there (mayo).

Every programmer should understand locales even if they do not speak the language.

Re: How we spent $30k in Firebase in less than 72 hours

#29

Post by mickronome »

itcmcgrath wrote: post #11
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

I would very much like to have something like Google authenticator, but for billing. With the ability so set it as alarms on my phone (and coworkers), preferably with some smarts to detect short usage spikes.

In essence, settings and updating amount, rate and velocity ( speed of rate change ) caps on the fly.

Then I can set whatever tight limit I want to, and not worry about burning through too much cash because of some simple coding, or config error.

A bug almost cost us several tens of thousands of BigQuery costs when a dev accidentally repeated a big query every 5 seconds in an automated script, and while we still had budget warnings, it still cost us a fair bit of money. Even after this, I found it tricky to set/catch budgets for single services. I think I had to use stackdriver to be able to get any kind of warning.

It was in the ’blinking lights and sirens’- territory fast!

Post reply on HN