Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

91–100 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#91

Post by cheeze »

itcmcgrath wrote: post #20
alasdair_ wrote: post #14

Earlier quoted context omitted.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

Urgh, that's terrible! That definitely shouldn't be the case. I'd love for you to send me any support case you had so I can review. Even if the front door of the system didn't help you, we definitely shouldn't have been able to get you in a good state much quicker. My profile has Twitter and my DMs are open (I can give you my email there too). Doing my kids dinner, so responses might be slightly delayed.

This is exactly why I can't trust Google services though. The services are rock solid, but far far too often do I see folks who have to reach out to a dev on Twitter or their friend at Google to get a simple billing error resolved correctly.

Its a problem with most cloud providers, but Google seems to be notorious for it.

Re: How we spent $30k in Firebase in less than 72 hours

#92

Post by Xorlev »

vbezhenar wrote: post #87
everdev wrote: post #4

Earlier quoted context omitted.

Even with a cap, a rogue line or a legitimate surge in traffic could shut down your app. It's endless bill monitoring and budget approval. I'll stick to a flat rate DO droplet.

Is it possible to make a similar approach in cloud? Something like "if traffic goes up, scale up until $5/month and then don't scale further, let it be slow (or even better throw errors)". Would be the best of both worlds.

For services that are pay per request, this is effectively a hard cap.

For more capacity based services, sure, but it's more likely to be down than just slow. When systems are run at their limit they rarely operate the way they did with a little less traffic.

Re: How we spent $30k in Firebase in less than 72 hours

#93

Post by boulos »

Disclosure: I work on Google Cloud (but not Firestore or Firebase).

For those that didn't read the article, it had a happy ending:

> GOOGLE UNDERSTOOD AND POWER US UP!

> After we fixed this code mistake, and stopped the billing, we reached out to Google to let them know the case and to see if we could apply for the next grant they have for startups. We told them that we spent the full 25k grant we had just a few days ago and see the chance to apply for the 100k grant on Google Cloud Services. We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant.

> Now we could not be more grateful to Google, not only for having an awesome “Backend As A Service” like Firebase, but also for letting us have 2 million sessions, 60 supports per minute and billions of requests without letting our site go down. Besides they understood errors like ours can happen when a startup is growing and some expensive mistakes can jeopardize the future great companies.

Re: How we spent $30k in Firebase in less than 72 hours

#94

Post by Dylan16807 »

hayleox wrote: post #78

This reads less like a "mistake", and more like just not giving any thought to what your code is doing as you write it. A mistake would be if the programmer meant to implement X but instead implemented Y; this sounds more like the programmer just set out to implement Y without even considering other possibilities. Same for the framework change. An outdated framework might be a second or so slower, but a >30-second lo…

They calculated a number the wrong way. That's not a hard mistake to make. And changing frameworks is silly but not really part of the problem.

Re: How we spent $30k in Firebase in less than 72 hours

#95

Post by Xorlev »

diziet wrote: post #79
ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

It's only 460k QPS, with 16k documents everything would be cached really well. Or a single instance of redis can serve that load of reads fairly easily.

Good luck with that. A single Redis would not be able to serve that workload. Maybe a single machine but you're really pushing the limits there just with concurrent TCP connections.

At that qps Redis has 2 microseconds per request.

I agree it caches well but your proposed architecture is definitely not production quality.

Re: How we spent $30k in Firebase in less than 72 hours

#97

Post by kylec »

itcmcgrath wrote: post #11
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

Are the caps on by default? If not, they should be.

Re: How we spent $30k in Firebase in less than 72 hours

#98

Post by Dylan16807 »

badlucklottery wrote: post #68
paranoidrobot wrote: post #21

Earlier quoted context omitted.

This seems like an overly cynical/snarky response. It's not an unreasonable request that for services which advertise the ability to scale up and down on demand, that the billing and billing limits should also be able to respond similarly.

>that the billing and billing limits should also be able to respond similarly How so? With a pay-as-you-go system, firing off warnings and giving a projection of their future costs (which is hard when startups tend to have spikey traffic) is about as good as you can do. Edit: I should add that the common solution to controlling your billing in situations like this is having some overflow path built into your beta app…

The problem is that the user gets those alerts, tries to change the limit they set, and it doesn't work. That is definitely not as good as can be done.

Re: How we spent $30k in Firebase in less than 72 hours

#100

Post by Spivak »

RobertRoberts wrote: post #64

Anyone else feel like they'd rather have their dedicated server slow down instead of wrack up a $30k debt? This is this nightmare I envisioned with cloud services, a client gets hit really hard, and I have to pass the bill on to them. This reminds me variable rate mortgages. With dedicated hardware, you may end up with performance issues, but never a ghastly business-ending bill. How does anyone justify this risk? I…

You use dedicated hardware with manual or semi-automated scaling until you're big enough to want unlimited and green enough to eat a large bill knowing that you still made a profit.
Post reply on HN