Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we spent $30k in Firebase in less than 72 hours

hackernoon.com

Post reply on HN
Print view View original

211–220 of 249 posts

Re: How we spent $30k in Firebase in less than 72 hours

#211

Post by pknopf »

sciurus wrote: post #37

When you have an unexplained performance problem, your response shouldn't be to "upgrade every single framework and plugin" that you use. The 36 hours that they spent doing this cost them $21,600 dollars on GCP and didn't solve their users' problem. Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system a…

> Understand the services you depend on. Track the number of requests you're making to them, how long they're taking, and how many are failing. Reason through your system and look at the data when you have issues, rather than grasping at straws.

This is a good thought-process even when debugging issues during development. I've seen many developers attempt to "fix" issues by trying to figure out what dance/keystroke makes things work.

Whenever you encounter an issue of any kind, anywhere, understand the issue before attempting to resolve it. It may require you to dig deep into things you don't currently understand, but your career is currently telling you that you need to understand it.

Re: How we spent $30k in Firebase in less than 72 hours

#212

Post by macspoofing »

toast0 wrote: post #103
thatjsguy wrote: post #63

Earlier quoted context omitted.

I mean this in the nicest possible way: I see this all the time with the JavaScript set and I am absolutely not the least bit surprised. I used to work on a team that was TypeScript top to bottom, with people who didn’t really even understand how to debug (they were mostly bootcamp juniors). Whenever something would break, if restating it didn’t work, you know what they’d try? Yup, upgrading random dependencies. Refa…

So I completely agree, but let me play devil's advocate. If you do go and track down the problem in your depedency and file a bug, one of two things is likely to happen: they close it and say it's fixed in the latest version or they refuse to accept your bug because it's filed against an old version. Skipping the track it down part and just jumping into upgrading can be a time saver. It works fairly well if you fit i…

>If you do go and track down the problem in your depedency and file a bug

There's a difference between upgrading your dependencies because you traced a problem that you know is fixed in the newer version and upgrading your dependencies because you hope it fixes a problem you don't understand.

Re: How we spent $30k in Firebase in less than 72 hours

#213

Post by xtrapolate »

Simple stress tests should've revealed this. Basic profiling should've revealed this. This article makes it appear that they've went live without ever really testing the infrastructure under any kind of load whatsoever.

The article refers to some mysterious "engineering team". It would appear very little actual engineering took place at that company.

Re: How we spent $30k in Firebase in less than 72 hours

#214

Post by snowwolf »

shusson wrote: post #189

| How we spent $30k in Firebase in less than 72 hours By not checking Google Billing after you launch your website. At the very least you should have a billing alert.

At the very least Google (and really all cloud services) could have reasonable default billing alerts built in. And automatic spike detection.

I'd rather have default alerts already configured that I can change, rather than none.

Re: How we spent $30k in Firebase in less than 72 hours

#215

Post by overcast »

macspoofing wrote: post #210
overcast wrote: post #200

Earlier quoted context omitted.

There has been MANY times, upgrading outdated libraries or software versions resolved similar performance issues for me. Should you just immediately run update all? No. But acting like that's not a viable solution is silly.

>But acting like that's not a viable solution is silly. Usually you want to understand the problem before solving it. In this case, they wasted a bunch of time doing a bunch of things (upgrading all the dependencies, and refactoring the app) in the hope that something (ANYTHING) they're doing hopefully fixes a problem they don't understand. Smart move?

Everyone can sit back in hindsight and act like armchair developers. When you're in the shit, sometimes you make not so smart moves, now they've learned from their mistakes. Which is why they are talking about it.

Re: How we spent $30k in Firebase in less than 72 hours

#216

Post by manishsharan »

The issue with GCP and AWS providing limitless scaleability is that bad code gets a free ride and we all occasionally write bad code. If the OP had tested this on a database with 1 CPU server with 2GB RAM, they would have caught it quickly in early dev testing.

Re: How we spent $30k in Firebase in less than 72 hours

#217

Post by snowwolf »

itcmcgrath wrote: post #11
t0mbstone wrote: post #3

This is why infinitely scaling pay-as-you-go cloud services terrify me. I refuse to use a service like this unless it gives me the ability to automatically cap costs and alert me when thresholds are met. All it takes is a rogue line of code in an endless loop or something, and you are bankrupt. Their site seems pretty basic. I'm struggling to understand why they couldn't just run it with something like Postgres for l…

Product Manager for Cloud Firestore here. It's worth noting we do have the ability to set hard daily caps, as well as budgets that can have alerts tied to them. It's something we also looking at ways to improve it.

How about having default conservative alerts built in for new accounts (not caps, just alerts). That way people who forget to set them will be reminded the first time they get one.

An account that goes from $0 spend to $30k in 72 hours should really trigger some kind of flag - even internally within google. What if they didn't have any kind of grant and weren't able to pay?

Re: How we spent $30k in Firebase in less than 72 hours

#218

Post by joeblau »

The thing I'm wondering about with the seniority of the dev team and what the code commit and code review process is. It seems like with a senior dev reviewing commits, someone would have caught that redundant work was being done.

I worked at a startup in the Mission for a few months and I remember seeing a quadratic query that ran for every customer that was logged into our application. The CEO and team lead wondered why our app worked great in Dev (with only 5 users) but was terrible on premise (250 users). When I tried to explain the issue the two devs before me didn't really understand what I was talking about. It was a quick refactor and caching solution that fixed the problem, but it was clear that the development was still new.

Re: How we spent $30k in Firebase in less than 72 hours

#219

Post by jasonkester »

alexpi wrote: post #174
paradite wrote: post #45

So it's a classic N+1 query?

It's more like (in good old RDBMS world) 'SELECT * FROM payments' and calculate sum on client vs 'SELECT SUM(amount) FROM payments WHERE some_id = ?' though they already had this sum precalculated

That would be more forgivable. This is:

  SELECT * FROM payments where paymentID = 1
  SELECT * FROM payments where paymentID = 2
  SELECT * FROM payments where paymentID = 3
  ...
  SELECT * FROM payments where paymentID = 14986
Each of those in its own API request over the wire, then sum them on the client.
Post reply on HN