Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

141–150 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#141

Post by betadreamer »

A lot of people here talks about lack of load testing and other "do it the right way" type of advice, but remember this is a startup. In my opinion, solid testing foundation will be such an overkill and the time is better spent on implementing more features.

Also I bet they did some manual testing. They didn't catch it because this latency can only be seen by the account with a lot of followers.

I agree that their first solution to upgrade is a bad idea... You should understand what caused the bug before trying to fix it.

I highly encourage you to monitor the load/pay/request graph on a daily basis. Even better if you hang a screen on the office that displays these. The graphs are already provided by Firebase. That way you can catch these type of anomaly on day one. Also Firebase supports "Progressively roll out new features" https://firebase.google.com/use-cases/#new-features

Re: How we spent $30k in Firebase in less than 72 hours

#142

Post by asadjb »

onion2k wrote: post #123
alasdair_ wrote: post #14

Earlier quoted context omitted.

>Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it. Google Cloud user here. A warning: If you ever happen to get, say, frontpage on reddit or techcrunch or other big boost to publicity, your site could be down until the next billing cycle (i.e. 24 hours) a…

Your app shouldn't just fail if an API is unavailable. You should code things to fail gracefully. In the case of a traffic spike replacing the front end with an email capture form and a message saying "things are really popular right now, how about we remind you about this tomorrow (with a discount code!)" works well. This is hard for some things, but your startup failing because you didn't want to do it is much hard…

Most applications have atleast 1 SPOF, the DB. Just because the DB is now the Firebase API doesn't change anything.

It would be very difficult to build products in a reasonable amount of time if everything has to be coded defensively. I can build my app quicker, and remain sane, if I assume that the DB will always be available, and just fail if the DB isn't there. Same for things like S3 (which I think had only 1 large scale failure in recent history), Redis, etc...

There are APIs which can be unavailable and you need to work around those. For me, these are mostly third party services that I don't control. But then again, I'm not building the next Netflix. I don't have enough engineers to build an app that works with a chaos monkey!

Not the best approach for all applications, but has been good enough for most projects I've worked on. Just my 2 cents.

Re: How we spent $30k in Firebase in less than 72 hours

#144

Post by bartread »

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant."

Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work together to achieve a bunch of shared goals and doing an imperfect job of it. This isn't just rose-tinted: I'm quite sure they have their fair share of bad actors, and they certainly make decisions we don't all like (e.g., retiring products), but I don't think it's because the company is fundamentally evil.

Re: How we spent $30k in Firebase in less than 72 hours

#145

Post by kompiuter »

ConcernedCoder wrote: post #54

"This means that every session to our site read the same number of documents as we have of number of payments. #UnaVacaPorDeLaCalle received more than 16,000 supporters, so: 2 million sessions x 16,000 documents = more than 40 Billion requests to Firestore on less than 48 hours." TLDR; Horrible architecture decisions like this can be very costly.

Legit question here; what would be a good architecture for this case?

Re: How we spent $30k in Firebase in less than 72 hours

#146

Post by cm2187 »

bartread wrote: post #144

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work togethe…

Except that their core business is to build files on everyone of their users to sell ads...

We could have a debate on whether Philip Morris is evil. I am sure most of their employees are pretty decent people.

Re: How we spent $30k in Firebase in less than 72 hours

#147

Post by Aloha »

flukus wrote: post #107
Aloha wrote: post #67

Earlier quoted context omitted.

Try-something is very useful as a troubleshooting tool, when you need to change the state of the issue enough to collect further troubleshooting information.

I think "try-anything" would be a better characterization in this case. They had smoke pouring out of the engine and tried changing the tire.

No disagreement there.

I don't understand how you can build a complex application like that without doing basic performance checks like, are we hitting the file system or database too often, our the image assets correctly sized, etc.

I'm not a software engineer however.

Re: How we spent $30k in Firebase in less than 72 hours

#148

Post by rafael859 »

bartread wrote: post #144

"We contacted the team of Google Developers Latam, to tell them what had just happened. They allowed us to apply for the next grant, which google approved, and after some meetings with them, they let us pay our bill with the grant." Makes for an interesting counterpoint to the currently popular "Google is evil" narrative. The truth is probably much more mundane: Google is an awful lot of people trying to work togethe…

I agree with your argument, but I don't think that people are calling Google evil because they retired some products. See [0], which was resolved (I think, though I can't find a source for that) due to the fact that Google is comprised of a lot of different people, and [1], which is a fairly recent announcement that has made some people uncomfortable.

[0] https://news.ycombinator.com/item?id=17202179

[1] https://news.ycombinator.com/item?id=17660872

Re: How we spent $30k in Firebase in less than 72 hours

#149

Post by cm2187 »

TeMPOraL wrote: post #116
sitkack wrote: post #40

Earlier quoted context omitted.

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is…

There is also a common sense problem. Not trying to play the blame game but the guy who ran a subtotal displayed on the front page by having to download every row in the database probably isn’t a great programmer. It’s easy to say don’t write bugs, but there are bugs and moronic design decisions. This to me doesn’t feel like a real bug.

Re: How we spent $30k in Firebase in less than 72 hours

#150

Post by voltagex_ »

TeMPOraL wrote: post #116
sitkack wrote: post #40

Earlier quoted context omitted.

> Reason through your system modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. abstractions make reasoning harder, what tools and techniques would you suggest for doing this? I'd probably run the application in some sort of sandbox and measure the outbound request load vs inbound request load, something a containerized deployment should be giving the end user (developer) as an…

> modern stacks have a huge opacity problem, everyone wants to be magic, and everyone fails. Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers, yes, but they aren't hard to pry apart. But people want immediate results without doing necessary cognitive work - understanding-guided exploration. It usually isn't hard to identify which component of your product is…

>Also, lots of people are impatient and/or intellectually lazy. We have piled up a ton of abstraction layers

Sorry, the guy that wrote/implemented abstraction layers 2 and 3 left 2 years ago and didn't document anything.

We've been understaffed for a year and we've been told not to hire any more staff until the new financial year.

I've got a "technical debt" item on the backlog but business drives the priorities and it'll never get done.

Post reply on HN